00010000:00010000:2.0:1787695648.385622:0:98780:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.385633:0:98780:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.385637:0:98780:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.385640:0:98780:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328a59 00000080:00010000:2.0:1787695648.385644:0:98780:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a59 from it ffffb5d403febce0 00010000:00010000:2.0:1787695648.385647:0:98780:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.385655:0:98780:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.385663:0:98780:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.385670:0:98780:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.385677:0:98780:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.385683:0:98780:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.385691:0:98780:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.385698:0:98780:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.385702:0:98780:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.385705:0:98780:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328a59 00000080:00010000:2.0:1787695648.385708:0:98780:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a59 from it ffffb5d403febdc8 00010000:00010000:2.0:1787695648.385711:0:98780:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.385722:0:98780:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695648.385730:0:98780:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.385735:0:98780:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695648.385765:0:98780:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.385806:0:98780:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.385813:0:98780:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695648.385816:0:98780:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328a59 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ecfd7 expref: -99 pid: 98780 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695648.385840:0:98780:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.385846:0:98780:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f9880 x1874531133991296/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.386221:0:98780:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98780:1874531133991296:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695648.386309:0:98780:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.392921:0:98780:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f9880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98780:1874531133991296:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695648.392972:0:98780:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x4f8:0x0](ffff9d85b3cc00c8) 00000080:00200000:2.0:1787695648.392980:0:98780:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x4f8:0x0], updating i_size 512 00000100:00100000:2.0:1787695648.446570:0:98782:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b73b0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98782:1874531133992576:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695648.446622:0:98782:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x4f9:0x0](ffff9d85b3cc0908) 00000080:00200000:2.0:1787695648.446631:0:98782:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x4f9:0x0], updating i_size 512 00000080:00200000:2.0:1787695648.452612:0:98783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.452703:0:98783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.452717:0:98783:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.452776:0:98783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98783 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.452787:0:98783:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98783 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.452798:0:98783:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98783 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.452824:0:98783:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fa680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98783:1874531133992704:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.453186:0:98783:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.455661:0:98783:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fa680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98783:1874531133992704:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.455688:0:98783:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455700:0:98783:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455712:0:98783:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455723:0:98783:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455734:0:98783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455742:0:98783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455749:0:98783:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.455761:0:98783:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fa680 x1874531133992704/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.455775:0:98783:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455789:0:98783:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.455795:0:98783:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.455827:0:98783:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.455832:0:98783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328a75 00000080:00010000:2.0:1787695648.455839:0:98783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.455842:0:98783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a75 from it ffffb5d403d77c20 00010000:00010000:2.0:1787695648.455845:0:98783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.455902:0:98783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.455927:0:98783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.455948:0:98783:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.455964:0:98783:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.455984:0:98783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.456037:0:98783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98783 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.456044:0:98783:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98783 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.456053:0:98783:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98783 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.456073:0:98783:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fa680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98783:1874531133992832:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.456146:0:98783:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.463661:0:98783:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fa680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98783:1874531133992832:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.463688:0:98783:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463711:0:98783:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 4/1,0 mode: --/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463719:0:98783:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 4/1,0 mode: --/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463730:0:98783:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463741:0:98783:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463753:0:98783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463762:0:98783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463770:0:98783:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.463781:0:98783:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fa680 x1874531133992832/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.463794:0:98783:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463804:0:98783:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1273 (0 0) 00000080:00200000:2.0:1787695648.463810:0:98783:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695648.463840:0:98783:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x4f9:0x0](ffff9d85b3cc1148) 00000080:00200000:2.0:1787695648.463878:0:98783:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x4f9:0x0], updating i_size 512 00000080:00010000:2.0:1787695648.463887:0:98783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x4f9:0x0](ffff9d85b3cc1148) for lock 0x678ae32e95328a7c 00000080:00010000:2.0:1787695648.463899:0:98783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x4f9:0x0](ffff9d85b3cc1148) 00000080:00010000:2.0:1787695648.463903:0:98783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a7c from it ffffb5d403d77e08 00010000:00010000:2.0:1787695648.463907:0:98783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.463917:0:98783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.463963:0:98783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.463982:0:98783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.463992:0:98783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.464005:0:98783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.464202:0:98783:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.464209:0:98783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.464212:0:98783:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328a75 00000080:00010000:2.0:1787695648.464217:0:98783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a75 from it ffffb5d403d77d90 00010000:00010000:2.0:1787695648.464220:0:98783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.464228:0:98783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.464237:0:98783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.464245:0:98783:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695648.464264:0:98783:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.464412:0:98783:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.464457:0:98783:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 1/0,0 mode: --/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.464465:0:98783:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.464477:0:98783:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695648.464479:0:98783:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a7c lrc: 0/0,0 mode: --/PR res: [0x200000401:0x4f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed06a expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.464509:0:98783:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a75 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed04e expref: -99 pid: 98783 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695648.464525:0:98783:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.464533:0:98783:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fa680 x1874531133993088/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.464556:0:98783:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fa680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98783:1874531133993088:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695648.464690:0:98783:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.473473:0:98783:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fa680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98783:1874531133993088:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695648.473536:0:98783:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x4f9:0x0](ffff9d85b3cc1148) 00000080:00200000:2.0:1787695648.511542:0:98785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.511571:0:98785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.511586:0:98785:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.511647:0:98785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.511657:0:98785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.511666:0:98785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.511691:0:98785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98785:1874531133993600:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.511819:0:98785:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.515841:0:98785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f9500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98785:1874531133993600:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.516153:0:98785:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516167:0:98785:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516178:0:98785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516188:0:98785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516200:0:98785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516208:0:98785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516216:0:98785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.516228:0:98785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f9500 x1874531133993600/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.516241:0:98785:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516259:0:98785:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.516266:0:98785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.516352:0:98785:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.516359:0:98785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328a8a 00000080:00010000:2.0:1787695648.516364:0:98785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.516367:0:98785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a8a from it ffffb5d404003c20 00010000:00010000:2.0:1787695648.516370:0:98785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516380:0:98785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.516402:0:98785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.516423:0:98785:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.516428:0:98785:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.516446:0:98785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.516482:0:98785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516490:0:98785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.516510:0:98785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.516528:0:98785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98785:1874531133993728:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.516627:0:98785:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.521553:0:98785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f9500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98785:1874531133993728:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.521584:0:98785:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521608:0:98785:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521618:0:98785:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521698:0:98785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521712:0:98785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521727:0:98785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521736:0:98785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521743:0:98785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.521756:0:98785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f9500 x1874531133993728/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.521771:0:98785:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.521783:0:98785:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1274 (0 0) 00000080:00200000:2.0:1787695648.521790:0:98785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695648.521827:0:98785:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x4fa:0x0](ffff9d85b3cc1148) 00000080:00200000:2.0:1787695648.521836:0:98785:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x4fa:0x0], updating i_size 512 00000080:00010000:2.0:1787695648.521843:0:98785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x4fa:0x0](ffff9d85b3cc1148) for lock 0x678ae32e95328a91 00000080:00010000:2.0:1787695648.522173:0:98785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x4fa:0x0](ffff9d85b3cc1148) 00000080:00010000:2.0:1787695648.522179:0:98785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a91 from it ffffb5d404003e08 00010000:00010000:2.0:1787695648.522184:0:98785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.522194:0:98785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.522225:0:98785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.522243:0:98785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.522253:0:98785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.522263:0:98785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.522269:0:98785:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.522273:0:98785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.522276:0:98785:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328a8a 00000080:00010000:2.0:1787695648.522351:0:98785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328a8a from it ffffb5d404003d90 00010000:00010000:2.0:1787695648.522355:0:98785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.522364:0:98785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.522374:0:98785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.522381:0:98785:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695648.522401:0:98785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.522434:0:98785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.522467:0:98785:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.522474:0:98785:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.522480:0:98785:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695648.522483:0:98785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328a91 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x4fa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed0c5 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.522509:0:98785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328a8a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed0a9 expref: -99 pid: 98785 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695648.522526:0:98785:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.522534:0:98785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f9500 x1874531133993984/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.522561:0:98785:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98785:1874531133993984:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695648.522718:0:98785:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.529867:0:98785:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f9500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98785:1874531133993984:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695648.529913:0:98785:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x4fa:0x0](ffff9d85b3cc1148) 00000080:00200000:2.0:1787695648.615457:0:98789:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.615486:0:98789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.620812:0:98789:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.620907:0:98789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.620918:0:98789:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.620928:0:98789:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98789 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.620955:0:98789:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fbb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98789:1874531133995392:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.622873:0:98789:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.626955:0:98789:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fbb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98789:1874531133995392:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.626983:0:98789:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.626996:0:98789:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627008:0:98789:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627020:0:98789:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627032:0:98789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627040:0:98789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627048:0:98789:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.627073:0:98789:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fbb80 x1874531133995392/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.627088:0:98789:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627102:0:98789:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.627109:0:98789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.627146:0:98789:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.627150:0:98789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ab4 00000080:00010000:2.0:1787695648.627153:0:98789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.627156:0:98789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ab4 from it ffffb5d40401bc20 00010000:00010000:2.0:1787695648.627158:0:98789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627167:0:98789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.627187:0:98789:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.627204:0:98789:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.627207:0:98789:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.627223:0:98789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.627260:0:98789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627265:0:98789:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.627271:0:98789:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98789 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.627285:0:98789:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fbb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98789:1874531133995520:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.627345:0:98789:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.631426:0:98789:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fbb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98789:1874531133995520:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.631448:0:98789:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631465:0:98789:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631471:0:98789:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631479:0:98789:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631486:0:98789:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631508:0:98789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631517:0:98789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631528:0:98789:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.631539:0:98789:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fbb80 x1874531133995520/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.631552:0:98789:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631562:0:98789:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1276 (0 0) 00000080:00200000:2.0:1787695648.631569:0:98789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695648.631603:0:98789:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x4fc:0x0](ffff9d85b3cc1148) 00000080:00200000:2.0:1787695648.631612:0:98789:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x4fc:0x0], updating i_size 512 00000080:00010000:2.0:1787695648.631618:0:98789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x4fc:0x0](ffff9d85b3cc1148) for lock 0x678ae32e95328abb 00000080:00010000:2.0:1787695648.631628:0:98789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x4fc:0x0](ffff9d85b3cc1148) 00000080:00010000:2.0:1787695648.631632:0:98789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328abb from it ffffb5d40401be08 00010000:00010000:2.0:1787695648.631635:0:98789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.631644:0:98789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.631675:0:98789:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.631690:0:98789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.631700:0:98789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.631710:0:98789:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.631717:0:98789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.633587:0:98789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.633598:0:98789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ab4 00000080:00010000:2.0:1787695648.633608:0:98789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ab4 from it ffffb5d40401bd90 00010000:00010000:2.0:1787695648.633613:0:98789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.633624:0:98789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.633642:0:98789:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.633652:0:98789:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695648.633684:0:98789:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.633725:0:98789:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.633768:0:98789:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 1/0,0 mode: --/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.633776:0:98789:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.633781:0:98789:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695648.633783:0:98789:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328abb lrc: 0/0,0 mode: --/PR res: [0x200000401:0x4fc:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed17b expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.633801:0:98789:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328ab4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed15f expref: -99 pid: 98789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695648.633817:0:98789:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.634557:0:98789:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fbb80 x1874531133995776/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.634589:0:98789:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fbb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98789:1874531133995776:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695648.635082:0:98789:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.639583:0:98789:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fbb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98789:1874531133995776:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695648.639624:0:98789:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x4fc:0x0](ffff9d85b3cc1148) 00000100:00100000:2.0:1787695648.660583:0:98790:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98790:1874531133996160:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695648.660796:0:98790:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x4fd:0x0](ffff9d85b3cc1148) 00000080:00200000:2.0:1787695648.660805:0:98790:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x4fd:0x0], updating i_size 512 00000080:00200000:2.0:1787695648.707921:0:98792:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.707957:0:98792:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.708048:0:98792:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98792 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.708061:0:98792:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98792 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.708071:0:98792:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98792 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.708098:0:98792:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72faa00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98792:1874531133996800:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695648.708182:0:98792:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.711811:0:98792:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72faa00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98792:1874531133996800:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695648.711838:0:98792:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.711884:0:98792:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.711898:0:98792:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.711908:0:98792:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.711920:0:98792:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.711932:0:98792:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.711939:0:98792:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.711961:0:98792:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72faa00 x1874531133996800/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.711973:0:98792:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.711985:0:98792:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.711991:0:98792:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.712092:0:98792:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.712100:0:98792:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ad7 00000080:00010000:2.0:1787695648.712105:0:98792:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.712108:0:98792:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ad7 from it ffffb5d40402bbe0 00010000:00010000:2.0:1787695648.712112:0:98792:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.712121:0:98792:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.712143:0:98792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.712164:0:98792:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695648.712168:0:98792:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.712182:0:98792:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.712191:0:98792:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.712200:0:98792:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.712206:0:98792:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.712210:0:98792:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.712213:0:98792:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ad7 00000080:00010000:2.0:1787695648.712216:0:98792:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ad7 from it ffffb5d40402bce0 00010000:00010000:2.0:1787695648.712218:0:98792:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.712225:0:98792:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.712233:0:98792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.712239:0:98792:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.712245:0:98792:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.712251:0:98792:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.712259:0:98792:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.712265:0:98792:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.712269:0:98792:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.712271:0:98792:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ad7 00000080:00010000:2.0:1787695648.712275:0:98792:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ad7 from it ffffb5d40402bdc8 00010000:00010000:2.0:1787695648.712278:0:98792:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.712284:0:98792:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695648.712291:0:98792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.712296:0:98792:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695648.712326:0:98792:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.712367:0:98792:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.712373:0:98792:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695648.712376:0:98792:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ad7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed1f9 expref: -99 pid: 98792 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695648.712400:0:98792:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.712407:0:98792:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72faa00 x1874531133997056/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.712429:0:98792:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72faa00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98792:1874531133997056:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695648.712508:0:98792:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.760417:0:98794:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbc380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98794:1874531133997952:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695648.760468:0:98794:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x4ff:0x0](ffff9d85b3cc7448) 00000080:00200000:2.0:1787695648.760475:0:98794:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x4ff:0x0], updating i_size 512 00000080:00200000:2.0:1787695648.768747:0:98795:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.768774:0:98795:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.768789:0:98795:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.768848:0:98795:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98795 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.769150:0:98795:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98795 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.769162:0:98795:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98795 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.769188:0:98795:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98795:1874531133998080:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.769328:0:98795:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.773429:0:98795:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98795:1874531133998080:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.773461:0:98795:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773474:0:98795:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773486:0:98795:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773507:0:98795:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773519:0:98795:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773530:0:98795:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773537:0:98795:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.773547:0:98795:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f8700 x1874531133998080/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.773559:0:98795:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773572:0:98795:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.773578:0:98795:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.773606:0:98795:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.773612:0:98795:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328af3 00000080:00010000:2.0:1787695648.773617:0:98795:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.773620:0:98795:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328af3 from it ffffb5d403ff3c20 00010000:00010000:2.0:1787695648.773639:0:98795:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773648:0:98795:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328af3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed270 expref: -99 pid: 98795 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.773672:0:98795:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.773698:0:98795:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.773703:0:98795:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.773723:0:98795:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.773762:0:98795:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328afa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98795 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773769:0:98795:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328afa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98795 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.773776:0:98795:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328afa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98795 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.773808:0:98795:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98795:1874531133998208:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.773903:0:98795:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000080:00200000:2.0:1787695648.784811:0:98795:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x4ff:0x0](ffff9d85b3ccc2c8) 00000100:00100000:2.0:1787695648.803506:0:98796:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbce00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98796:1874531133998848:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695648.803554:0:98796:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x500:0x0](ffff9d85b3cc3a88) 00000080:00200000:2.0:1787695648.803560:0:98796:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x500:0x0], updating i_size 512 00000080:00200000:2.0:1787695648.808777:0:98797:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.808805:0:98797:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.808820:0:98797:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.808906:0:98797:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98797 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.808916:0:98797:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98797 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.808938:0:98797:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98797 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.809164:0:98797:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fa300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98797:1874531133998976:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.810764:0:98797:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.814010:0:98797:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fa300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98797:1874531133998976:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.814099:0:98797:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814111:0:98797:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814122:0:98797:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814134:0:98797:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814146:0:98797:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814154:0:98797:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814161:0:98797:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.814174:0:98797:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fa300 x1874531133998976/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.814187:0:98797:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814201:0:98797:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.814209:0:98797:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.814234:0:98797:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.814241:0:98797:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b08 00000080:00010000:2.0:1787695648.814246:0:98797:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.814249:0:98797:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b08 from it ffffb5d403ff3c20 00010000:00010000:2.0:1787695648.814253:0:98797:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814261:0:98797:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.814284:0:98797:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.814307:0:98797:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.814311:0:98797:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.814330:0:98797:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.814371:0:98797:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98797 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814379:0:98797:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98797 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.814387:0:98797:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98797 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.814407:0:98797:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fa300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98797:1874531133999104:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.814531:0:98797:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.818522:0:98797:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fa300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98797:1874531133999104:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.818548:0:98797:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818569:0:98797:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 4/1,0 mode: --/PR res: [0x200000401:0x500:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818577:0:98797:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 4/1,0 mode: --/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818588:0:98797:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818598:0:98797:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818611:0:98797:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818619:0:98797:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818766:0:98797:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.818778:0:98797:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fa300 x1874531133999104/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.818791:0:98797:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.818804:0:98797:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1280 (0 0) 00000080:00200000:2.0:1787695648.818811:0:98797:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695648.818841:0:98797:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x500:0x0](ffff9d85b3cc1988) 00000080:00200000:2.0:1787695648.818917:0:98797:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x500:0x0], updating i_size 512 00000080:00010000:2.0:1787695648.818926:0:98797:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x500:0x0](ffff9d85b3cc1988) for lock 0x678ae32e95328b0f 00000080:00010000:2.0:1787695648.818938:0:98797:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x500:0x0](ffff9d85b3cc1988) 00000080:00010000:2.0:1787695648.819003:0:98797:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b0f from it ffffb5d403ff3e08 00010000:00010000:2.0:1787695648.819008:0:98797:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.819078:0:98797:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.819111:0:98797:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.819126:0:98797:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.819137:0:98797:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.819148:0:98797:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.819154:0:98797:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.819159:0:98797:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.819162:0:98797:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b08 00000080:00010000:2.0:1787695648.819166:0:98797:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b08 from it ffffb5d403ff3d90 00010000:00010000:2.0:1787695648.819168:0:98797:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.819177:0:98797:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.819185:0:98797:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.819192:0:98797:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695648.819211:0:98797:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.819242:0:98797:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.819278:0:98797:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 1/0,0 mode: --/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.819286:0:98797:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.819291:0:98797:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695648.819295:0:98797:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b0f lrc: 0/0,0 mode: --/PR res: [0x200000401:0x500:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed2e7 expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.819309:0:98797:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b08 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed2cb expref: -99 pid: 98797 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695648.819323:0:98797:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.819330:0:98797:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fa300 x1874531133999360/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.819357:0:98797:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fa300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98797:1874531133999360:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695648.819475:0:98797:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.826894:0:98797:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fa300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98797:1874531133999360:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695648.826950:0:98797:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x500:0x0](ffff9d85b3cc1988) 00000080:00200000:2.0:1787695648.835254:0:98798:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.835280:0:98798:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.835345:0:98798:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98798 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.835356:0:98798:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98798 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.835365:0:98798:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98798 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.835391:0:98798:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98798:1874531133999488:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695648.835542:0:98798:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.838504:0:98798:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f9c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98798:1874531133999488:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695648.838530:0:98798:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838542:0:98798:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838552:0:98798:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838566:0:98798:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838577:0:98798:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838584:0:98798:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838590:0:98798:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.838602:0:98798:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f9c00 x1874531133999488/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.838614:0:98798:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838688:0:98798:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.838696:0:98798:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.838723:0:98798:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.838729:0:98798:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b16 00000080:00010000:2.0:1787695648.838735:0:98798:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.838738:0:98798:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b16 from it ffffb5d403f87be0 00010000:00010000:2.0:1787695648.838742:0:98798:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.838756:0:98798:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.838778:0:98798:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.838803:0:98798:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695648.838807:0:98798:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.838821:0:98798:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.838829:0:98798:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.838838:0:98798:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.838844:0:98798:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.838926:0:98798:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.838930:0:98798:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b16 00000080:00010000:2.0:1787695648.838934:0:98798:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b16 from it ffffb5d403f87ce0 00010000:00010000:2.0:1787695648.838937:0:98798:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.839014:0:98798:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.839088:0:98798:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.839095:0:98798:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.839103:0:98798:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.839110:0:98798:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.839119:0:98798:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.839125:0:98798:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.839129:0:98798:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.839132:0:98798:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b16 00000080:00010000:2.0:1787695648.839135:0:98798:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b16 from it ffffb5d403f87dc8 00010000:00010000:2.0:1787695648.839138:0:98798:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.839144:0:98798:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695648.839152:0:98798:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.839157:0:98798:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695648.839187:0:98798:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.839231:0:98798:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.839239:0:98798:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695648.839242:0:98798:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b16 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed30a expref: -99 pid: 98798 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695648.839265:0:98798:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.839273:0:98798:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f9c00 x1874531133999744/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.839295:0:98798:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98798:1874531133999744:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695648.839418:0:98798:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000080:00200000:2.0:1787695648.854686:0:98799:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.854717:0:98799:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.854734:0:98799:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.854798:0:98799:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98799 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.854808:0:98799:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98799 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.854818:0:98799:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98799 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.854844:0:98799:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98799:1874531133999872:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.856191:0:98799:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.859258:0:98799:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98799:1874531133999872:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.859286:0:98799:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859298:0:98799:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859308:0:98799:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859323:0:98799:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859335:0:98799:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859344:0:98799:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859351:0:98799:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.859363:0:98799:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f8e00 x1874531133999872/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.859376:0:98799:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859388:0:98799:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.859395:0:98799:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.859421:0:98799:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.859428:0:98799:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b1d 00000080:00010000:2.0:1787695648.859433:0:98799:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.859436:0:98799:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b1d from it ffffb5d40402bc20 00010000:00010000:2.0:1787695648.859439:0:98799:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859453:0:98799:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328b1d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed326 expref: -99 pid: 98799 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.859476:0:98799:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.859510:0:98799:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.859515:0:98799:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.859533:0:98799:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.859570:0:98799:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b24 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98799 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859577:0:98799:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b24 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98799 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.859585:0:98799:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328b24 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98799 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.859604:0:98799:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98799:1874531134000000:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.859721:0:98799:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000080:00200000:2.0:1787695648.898240:0:98801:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.898268:0:98801:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.898283:0:98801:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.898335:0:98801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98801 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.898345:0:98801:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98801 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.898356:0:98801:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98801 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.898382:0:98801:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98801:1874531134000768:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.898525:0:98801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.902351:0:98801:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98801:1874531134000768:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.902376:0:98801:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902388:0:98801:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902399:0:98801:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902409:0:98801:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902424:0:98801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902433:0:98801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902440:0:98801:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.902453:0:98801:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fb480 x1874531134000768/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.902473:0:98801:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902496:0:98801:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.902504:0:98801:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.902532:0:98801:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.902538:0:98801:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b32 00000080:00010000:2.0:1787695648.902543:0:98801:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.902546:0:98801:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b32 from it ffffb5d40403bc20 00010000:00010000:2.0:1787695648.902550:0:98801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902559:0:98801:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.902584:0:98801:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.902608:0:98801:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.902613:0:98801:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.902691:0:98801:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.902738:0:98801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98801 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902746:0:98801:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98801 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.902754:0:98801:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98801 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.902776:0:98801:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98801:1874531134000896:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.903048:0:98801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.906538:0:98801:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98801:1874531134000896:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.906566:0:98801:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906589:0:98801:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x502:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906597:0:98801:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906608:0:98801:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906618:0:98801:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906641:0:98801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906649:0:98801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906657:0:98801:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.906676:0:98801:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fb480 x1874531134000896/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.906690:0:98801:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906701:0:98801:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1282 (0 0) 00000080:00200000:2.0:1787695648.906708:0:98801:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695648.906742:0:98801:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x502:0x0](ffff9d85b3cc1988) 00000080:00200000:2.0:1787695648.906751:0:98801:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x502:0x0], updating i_size 512 00000080:00010000:2.0:1787695648.906758:0:98801:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x502:0x0](ffff9d85b3cc1988) for lock 0x678ae32e95328b39 00000080:00010000:2.0:1787695648.906768:0:98801:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x502:0x0](ffff9d85b3cc1988) 00000080:00010000:2.0:1787695648.906772:0:98801:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b39 from it ffffb5d40403be08 00010000:00010000:2.0:1787695648.906775:0:98801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.906784:0:98801:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.906815:0:98801:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.906833:0:98801:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.906843:0:98801:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.907093:0:98801:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.907107:0:98801:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.907113:0:98801:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.907116:0:98801:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b32 00000080:00010000:2.0:1787695648.907120:0:98801:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b32 from it ffffb5d40403bd90 00010000:00010000:2.0:1787695648.907123:0:98801:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.907131:0:98801:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.907141:0:98801:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.907148:0:98801:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695648.907166:0:98801:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.907199:0:98801:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.907236:0:98801:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.907243:0:98801:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.907248:0:98801:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695648.907251:0:98801:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b39 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x502:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed39d expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.907271:0:98801:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b32 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed381 expref: -99 pid: 98801 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695648.907286:0:98801:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.907293:0:98801:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fb480 x1874531134001152/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.907316:0:98801:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98801:1874531134001152:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695648.907383:0:98801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.911299:0:98801:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98801:1874531134001152:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695648.911338:0:98801:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x502:0x0](ffff9d85b3cc1988) 00000080:00200000:2.0:1787695648.922293:0:98802:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.922319:0:98802:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.922374:0:98802:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98802 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.922384:0:98802:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98802 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.922394:0:98802:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98802 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.922416:0:98802:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98802:1874531134001280:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695648.922550:0:98802:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.926332:0:98802:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98802:1874531134001280:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695648.926360:0:98802:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926373:0:98802:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926384:0:98802:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926393:0:98802:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926405:0:98802:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926413:0:98802:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926419:0:98802:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.926429:0:98802:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f9f80 x1874531134001280/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.926442:0:98802:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926454:0:98802:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.926462:0:98802:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.926506:0:98802:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.926534:0:98802:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b40 00000080:00010000:2.0:1787695648.926540:0:98802:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.926543:0:98802:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b40 from it ffffb5d404043be0 00010000:00010000:2.0:1787695648.926546:0:98802:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.926554:0:98802:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.926577:0:98802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.926599:0:98802:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695648.926604:0:98802:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.926617:0:98802:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.926811:0:98802:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.926823:0:98802:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.926830:0:98802:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.926835:0:98802:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.926838:0:98802:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b40 00000080:00010000:2.0:1787695648.926843:0:98802:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b40 from it ffffb5d404043ce0 00010000:00010000:2.0:1787695648.926846:0:98802:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.927115:0:98802:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.927131:0:98802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.927139:0:98802:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.927147:0:98802:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.927154:0:98802:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.927163:0:98802:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.927169:0:98802:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.927173:0:98802:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.927175:0:98802:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b40 00000080:00010000:2.0:1787695648.927179:0:98802:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b40 from it ffffb5d404043dc8 00010000:00010000:2.0:1787695648.927182:0:98802:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.927189:0:98802:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695648.927197:0:98802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.927202:0:98802:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695648.927231:0:98802:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.927275:0:98802:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.927286:0:98802:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695648.927289:0:98802:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b40 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed3c0 expref: -99 pid: 98802 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695648.927311:0:98802:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.927318:0:98802:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f9f80 x1874531134001536/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.927339:0:98802:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f9f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98802:1874531134001536:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695648.927457:0:98802:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000080:00200000:2.0:1787695648.938131:0:98804:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695648.938159:0:98804:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.938173:0:98804:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695648.938231:0:98804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.938240:0:98804:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.938250:0:98804:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98804 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.938275:0:98804:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98804:1874531134001664:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.938411:0:98804:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.941412:0:98804:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98804:1874531134001664:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695648.941437:0:98804:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941454:0:98804:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941464:0:98804:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941474:0:98804:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941485:0:98804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941507:0:98804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941513:0:98804:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.941525:0:98804:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f8000 x1874531134001664/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.941538:0:98804:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941551:0:98804:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.941558:0:98804:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.941582:0:98804:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.941588:0:98804:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b47 00000080:00010000:2.0:1787695648.941594:0:98804:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695648.941596:0:98804:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b47 from it ffffb5d40404bc20 00010000:00010000:2.0:1787695648.941600:0:98804:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941608:0:98804:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328b47 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed3dc expref: -99 pid: 98804 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.941695:0:98804:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.941719:0:98804:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695648.941723:0:98804:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695648.941742:0:98804:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695648.941781:0:98804:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b4e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941789:0:98804:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b4e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98804 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.941797:0:98804:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328b4e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98804 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.941818:0:98804:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98804:1874531134001792:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695648.942133:0:98804:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000080:00200000:2.0:1787695648.965735:0:98805:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.965762:0:98805:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.965825:0:98805:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98805 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.965840:0:98805:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98805 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.966311:0:98805:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98805 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695648.966343:0:98805:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98805:1874531134002176:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695648.966508:0:98805:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.969784:0:98805:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98805:1874531134002176:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695648.969812:0:98805:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.969825:0:98805:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.969838:0:98805:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.970266:0:98805:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.970283:0:98805:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.970291:0:98805:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.970302:0:98805:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695648.970315:0:98805:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72f8380 x1874531134002176/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695665 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695648.970329:0:98805:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.970340:0:98805:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695648.970346:0:98805:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695648.970372:0:98805:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695648.970378:0:98805:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b55 00000080:00010000:2.0:1787695648.970382:0:98805:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.970385:0:98805:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b55 from it ffffb5d404053be0 00010000:00010000:2.0:1787695648.970388:0:98805:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695648.970396:0:98805:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.970420:0:98805:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695648.970441:0:98805:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695648.970446:0:98805:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.970460:0:98805:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.970469:0:98805:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.970483:0:98805:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695648.970502:0:98805:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.970507:0:98805:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.970509:0:98805:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b55 00000080:00010000:2.0:1787695648.970513:0:98805:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b55 from it ffffb5d404053ce0 00010000:00010000:2.0:1787695648.970515:0:98805:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695648.970522:0:98805:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.970530:0:98805:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.970536:0:98805:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695648.970543:0:98805:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695648.970549:0:98805:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.970556:0:98805:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695648.970561:0:98805:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695648.970565:0:98805:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695648.970567:0:98805:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328b55 00000080:00010000:2.0:1787695648.970570:0:98805:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b55 from it ffffb5d404053dc8 00010000:00010000:2.0:1787695648.970573:0:98805:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695648.970579:0:98805:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695648.970586:0:98805:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695648.970591:0:98805:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695648.970619:0:98805:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.970766:0:98805:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695648.970772:0:98805:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695648.970775:0:98805:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328b55 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed41b expref: -99 pid: 98805 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695648.970800:0:98805:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695648.970808:0:98805:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f8380 x1874531134002432/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695648.970830:0:98805:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98805:1874531134002432:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695648.971228:0:98805:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695648.976432:0:98805:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98805:1874531134002432:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695648.976481:0:98805:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x504:0x0](ffff9d85b3cc1988) 00000080:00200000:2.0:1787695648.976498:0:98805:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x504:0x0], updating i_size 512 00000100:00100000:2.0:1787695649.019652:0:98807:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babe2a00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98807:1874531134003328:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.019707:0:98807:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x505:0x0](ffff9d85b3cc2a08) 00000080:00200000:2.0:1787695649.019716:0:98807:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x505:0x0], updating i_size 512 00000100:00100000:2.0:1787695649.040747:0:98808:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babe3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98808:1874531134003584:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.040776:0:98808:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.040799:0:98808:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x505:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.040808:0:98808:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.040820:0:98808:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.040831:0:98808:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.040842:0:98808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.041356:0:98808:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.041366:0:98808:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.041381:0:98808:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babe3100 x1874531134003584/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.041394:0:98808:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.041405:0:98808:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1285 (0 0) 00000080:00200000:2.0:1787695649.041411:0:98808:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695649.041606:0:98808:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x505:0x0](ffff9d85b3cc5b88) 00000080:00200000:2.0:1787695649.041614:0:98808:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x505:0x0], updating i_size 512 00000080:00010000:2.0:1787695649.041622:0:98808:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x505:0x0](ffff9d85b3cc5b88) for lock 0x678ae32e95328b78 00000080:00010000:2.0:1787695649.041703:0:98808:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x505:0x0](ffff9d85b3cc5b88) 00000080:00010000:2.0:1787695649.041706:0:98808:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b78 from it ffffb5d40406be08 00010000:00010000:2.0:1787695649.041710:0:98808:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.041719:0:98808:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.041754:0:98808:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.041773:0:98808:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.041784:0:98808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e95328b71 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed492 expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.041794:0:98808:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e95328b71 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed492 expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.041799:0:98808:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.041804:0:98808:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.041808:0:98808:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b71 00000080:00010000:2.0:1787695649.041812:0:98808:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b71 from it ffffb5d40406bd90 00010000:00010000:2.0:1787695649.041815:0:98808:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e95328b71 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed492 expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.041821:0:98808:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e95328b71 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed492 expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.041829:0:98808:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.041837:0:98808:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695649.042121:0:98808:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e95328b71 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed492 expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.042169:0:98808:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.042219:0:98808:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.042226:0:98808:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e95328b71 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed492 expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.042232:0:98808:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.042235:0:98808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e95328b78 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x505:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed4ae expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.042251:0:98808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e95328b71 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed492 expref: -99 pid: 98808 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.042267:0:98808:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.042275:0:98808:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f8000 x1874531134003840/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.042298:0:98808:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98808:1874531134003840:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.042426:0:98808:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.048823:0:98808:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98808:1874531134003840:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.049824:0:98808:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x505:0x0](ffff9d85b3cc5b88) 00000080:00200000:2.0:1787695649.072556:0:98809:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x506:0x0](ffff9d85b3cc5b88) 00000080:00200000:2.0:1787695649.072564:0:98809:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x506:0x0], updating i_size 512 00000100:00100000:2.0:1787695649.086004:0:98810:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.087166:0:98810:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babe3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98810:1874531134004352:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.087191:0:98810:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.087203:0:98810:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.087215:0:98810:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.087225:0:98810:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.087238:0:98810:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.087246:0:98810:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.087254:0:98810:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.087267:0:98810:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babe3100 x1874531134004352/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.087281:0:98810:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328b86 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed4ed expref: -99 pid: 98810 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.087295:0:98810:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.087301:0:98810:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000100:00100000:2.0:1787695649.137263:0:98812:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98812:1874531134005376:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.137292:0:98812:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137313:0:98812:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x507:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137320:0:98812:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137330:0:98812:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137340:0:98812:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137357:0:98812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137364:0:98812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137372:0:98812:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.137384:0:98812:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3b800 x1874531134005376/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.137396:0:98812:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137405:0:98812:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1287 (0 0) 00000080:00200000:2.0:1787695649.137411:0:98812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695649.137467:0:98812:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x507:0x0](ffff9d85b3cddb88) 00000080:00200000:2.0:1787695649.137475:0:98812:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x507:0x0], updating i_size 512 00000080:00010000:2.0:1787695649.137482:0:98812:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x507:0x0](ffff9d85b3cddb88) for lock 0x678ae32e95328ba2 00000080:00010000:2.0:1787695649.137508:0:98812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x507:0x0](ffff9d85b3cddb88) 00000080:00010000:2.0:1787695649.137513:0:98812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ba2 from it ffffb5d403f87e08 00010000:00010000:2.0:1787695649.137516:0:98812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.137525:0:98812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.137560:0:98812:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.137576:0:98812:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.137585:0:98812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e95328b9b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed548 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.137594:0:98812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e95328b9b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed548 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.137600:0:98812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.137605:0:98812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.137608:0:98812:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328b9b 00000080:00010000:2.0:1787695649.137612:0:98812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328b9b from it ffffb5d403f87d90 00010000:00010000:2.0:1787695649.137615:0:98812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e95328b9b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed548 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.137622:0:98812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e95328b9b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed548 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.137698:0:98812:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.137706:0:98812:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695649.137726:0:98812:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e95328b9b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed548 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.137759:0:98812:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.137795:0:98812:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.137803:0:98812:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e95328b9b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed548 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.137809:0:98812:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.137811:0:98812:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e95328ba2 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x507:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed564 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.137829:0:98812:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e95328b9b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed548 expref: -99 pid: 98812 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.137844:0:98812:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.139151:0:98812:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f8e00 x1874531134005632/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.139187:0:98812:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98812:1874531134005632:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.139324:0:98812:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.145334:0:98812:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98812:1874531134005632:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.145380:0:98812:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x507:0x0](ffff9d85b3cddb88) 00000100:00100000:2.0:1787695649.160732:0:98813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98813:1874531134005760:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.160760:0:98813:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160774:0:98813:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160785:0:98813:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160796:0:98813:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160808:0:98813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160815:0:98813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160823:0:98813:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.160836:0:98813:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39c00 x1874531134005760/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.160877:0:98813:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160893:0:98813:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.160909:0:98813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.160946:0:98813:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.160954:0:98813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ba9 00000080:00010000:2.0:1787695649.160959:0:98813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.160963:0:98813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ba9 from it ffffb5d403d97be0 00010000:00010000:2.0:1787695649.160966:0:98813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.160980:0:98813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.161006:0:98813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.161040:0:98813:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.161045:0:98813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.161058:0:98813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.161067:0:98813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.161077:0:98813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.161082:0:98813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.161087:0:98813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.161089:0:98813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ba9 00000080:00010000:2.0:1787695649.161092:0:98813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ba9 from it ffffb5d403d97ce0 00010000:00010000:2.0:1787695649.161095:0:98813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.161101:0:98813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.161109:0:98813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.161115:0:98813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.161121:0:98813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.161127:0:98813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.161134:0:98813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.161140:0:98813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.161144:0:98813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.161146:0:98813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ba9 00000080:00010000:2.0:1787695649.161149:0:98813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ba9 from it ffffb5d403d97dc8 00010000:00010000:2.0:1787695649.161152:0:98813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.161159:0:98813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.161167:0:98813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.161172:0:98813:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.161204:0:98813:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.161255:0:98813:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.161263:0:98813:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.161266:0:98813:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e95328ba9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed587 expref: -99 pid: 98813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.161293:0:98813:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.161301:0:98813:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72f8700 x1874531134006016/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.161324:0:98813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72f8700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98813:1874531134006016:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.161405:0:98813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.165931:0:98813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72f8700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98813:1874531134006016:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.165991:0:98813:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x508:0x0](ffff9d85b3cdec08) 00000080:00200000:2.0:1787695649.165999:0:98813:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x508:0x0], updating i_size 512 00000080:00200000:2.0:1787695649.179567:0:98814:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.179600:0:98814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.179619:0:98814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695649.179807:0:98814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.179818:0:98814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.179829:0:98814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.180134:0:98814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98814:1874531134006144:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.180253:0:98814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.183835:0:98814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98814:1874531134006144:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.184046:0:98814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184060:0:98814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184075:0:98814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184086:0:98814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184099:0:98814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184107:0:98814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184115:0:98814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.184127:0:98814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fb100 x1874531134006144/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.184142:0:98814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184156:0:98814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.184162:0:98814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.184191:0:98814:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.184197:0:98814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328bb0 00000080:00010000:2.0:1787695649.184202:0:98814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.184205:0:98814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bb0 from it ffffb5d40404bc20 00010000:00010000:2.0:1787695649.184208:0:98814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184216:0:98814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.184240:0:98814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.184262:0:98814:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695649.184266:0:98814:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695649.184285:0:98814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695649.184328:0:98814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184335:0:98814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.184344:0:98814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.184364:0:98814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98814:1874531134006272:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.184498:0:98814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.188185:0:98814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98814:1874531134006272:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.188214:0:98814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188235:0:98814:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x508:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188243:0:98814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188254:0:98814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188264:0:98814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188275:0:98814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188283:0:98814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188290:0:98814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.188302:0:98814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fb100 x1874531134006272/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.188315:0:98814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188326:0:98814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1288 (0 0) 00000080:00200000:2.0:1787695649.188332:0:98814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695649.188365:0:98814:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x508:0x0](ffff9d85b3cddb88) 00000080:00200000:2.0:1787695649.188374:0:98814:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x508:0x0], updating i_size 512 00000080:00010000:2.0:1787695649.188380:0:98814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x508:0x0](ffff9d85b3cddb88) for lock 0x678ae32e95328bb7 00000080:00010000:2.0:1787695649.188390:0:98814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x508:0x0](ffff9d85b3cddb88) 00000080:00010000:2.0:1787695649.188394:0:98814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bb7 from it ffffb5d40404be08 00010000:00010000:2.0:1787695649.188397:0:98814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.188405:0:98814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.188433:0:98814:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.188447:0:98814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.188457:0:98814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.188468:0:98814:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.188474:0:98814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.188478:0:98814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.188491:0:98814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328bb0 00000080:00010000:2.0:1787695649.188495:0:98814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bb0 from it ffffb5d40404bd90 00010000:00010000:2.0:1787695649.188498:0:98814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.188505:0:98814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.188514:0:98814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.188521:0:98814:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695649.188541:0:98814:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.188572:0:98814:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.188605:0:98814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.188612:0:98814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.188618:0:98814:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.188620:0:98814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bb7 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x508:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed5bf expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.188637:0:98814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bb0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed5a3 expref: -99 pid: 98814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.188651:0:98814:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.188658:0:98814:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fb100 x1874531134006528/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.188678:0:98814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98814:1874531134006528:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.188751:0:98814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.194262:0:98814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98814:1874531134006528:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.194304:0:98814:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x508:0x0](ffff9d85b3cddb88) 00000080:00200000:2.0:1787695649.207248:0:98815:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.207277:0:98815:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.207338:0:98815:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98815 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.207349:0:98815:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98815 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.207358:0:98815:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98815 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.207383:0:98815:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98815:1874531134006656:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695649.207612:0:98815:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.216511:0:98815:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98815:1874531134006656:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.216545:0:98815:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216559:0:98815:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216570:0:98815:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216580:0:98815:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216592:0:98815:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216600:0:98815:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216611:0:98815:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.216623:0:98815:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fb800 x1874531134006656/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.216635:0:98815:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216647:0:98815:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.216653:0:98815:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.216683:0:98815:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.216691:0:98815:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328bbe 00000080:00010000:2.0:1787695649.216697:0:98815:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.216700:0:98815:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bbe from it ffffb5d40408bbe0 00010000:00010000:2.0:1787695649.216704:0:98815:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.216713:0:98815:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.216740:0:98815:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.216775:0:98815:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.216781:0:98815:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.216796:0:98815:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.216805:0:98815:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.216820:0:98815:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.216827:0:98815:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.216832:0:98815:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.216834:0:98815:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328bbe 00000080:00010000:2.0:1787695649.216838:0:98815:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bbe from it ffffb5d40408bce0 00010000:00010000:2.0:1787695649.216841:0:98815:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.216883:0:98815:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.217087:0:98815:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.217097:0:98815:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.217105:0:98815:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.217112:0:98815:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.217121:0:98815:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.217126:0:98815:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.217130:0:98815:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.217133:0:98815:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328bbe 00000080:00010000:2.0:1787695649.217136:0:98815:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bbe from it ffffb5d40408bdc8 00010000:00010000:2.0:1787695649.217140:0:98815:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.217146:0:98815:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.217155:0:98815:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.217161:0:98815:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.217198:0:98815:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.217254:0:98815:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.217262:0:98815:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.217265:0:98815:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328bbe lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed5e2 expref: -99 pid: 98815 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.217293:0:98815:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.217300:0:98815:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fb800 x1874531134006912/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.217323:0:98815:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fb800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98815:1874531134006912:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.217404:0:98815:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.222056:0:98815:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fb800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98815:1874531134006912:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.222123:0:98815:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x509:0x0](ffff9d85b3cdb248) 00000080:00200000:2.0:1787695649.222131:0:98815:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x509:0x0], updating i_size 512 00000100:00100000:2.0:1787695649.328420:0:98819:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98819:1874531134008448:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.328460:0:98819:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328474:0:98819:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328499:0:98819:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328511:0:98819:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328524:0:98819:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328532:0:98819:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328539:0:98819:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.328550:0:98819:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f38a80 x1874531134008448/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.328564:0:98819:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328576:0:98819:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.328583:0:98819:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.328618:0:98819:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.328624:0:98819:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328be8 00000080:00010000:2.0:1787695649.328629:0:98819:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.328632:0:98819:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328be8 from it ffffb5d403fe3be0 00010000:00010000:2.0:1787695649.328635:0:98819:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.328643:0:98819:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.328670:0:98819:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.328693:0:98819:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.328698:0:98819:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.328712:0:98819:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.328721:0:98819:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.328732:0:98819:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.328739:0:98819:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.328743:0:98819:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.328746:0:98819:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328be8 00000080:00010000:2.0:1787695649.328749:0:98819:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328be8 from it ffffb5d403fe3ce0 00010000:00010000:2.0:1787695649.328752:0:98819:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.328758:0:98819:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.328767:0:98819:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.328774:0:98819:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.328779:0:98819:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.328784:0:98819:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.328791:0:98819:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.328797:0:98819:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.328801:0:98819:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.328803:0:98819:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328be8 00000080:00010000:2.0:1787695649.328806:0:98819:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328be8 from it ffffb5d403fe3dc8 00010000:00010000:2.0:1787695649.328810:0:98819:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.328816:0:98819:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.328824:0:98819:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.328829:0:98819:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.328901:0:98819:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.328959:0:98819:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.328971:0:98819:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.328973:0:98819:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95328be8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed698 expref: -99 pid: 98819 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.328999:0:98819:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.329006:0:98819:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fad80 x1874531134008704/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.330252:0:98819:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fad80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98819:1874531134008704:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.330388:0:98819:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.337200:0:98819:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fad80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98819:1874531134008704:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.337261:0:98819:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x50b:0x0](ffff9d85b3cddb88) 00000080:00200000:2.0:1787695649.337271:0:98819:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x50b:0x0], updating i_size 512 00000080:00200000:2.0:1787695649.345470:0:98820:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.345516:0:98820:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.345532:0:98820:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695649.345604:0:98820:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98820 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.345613:0:98820:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98820 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.345793:0:98820:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98820 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.345823:0:98820:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98820:1874531134008832:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.346253:0:98820:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.349610:0:98820:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98820:1874531134008832:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.349652:0:98820:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349665:0:98820:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349677:0:98820:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349689:0:98820:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349701:0:98820:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349709:0:98820:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349716:0:98820:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.349728:0:98820:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab75880 x1874531134008832/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.349742:0:98820:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349755:0:98820:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.349763:0:98820:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.349795:0:98820:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.349801:0:98820:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328bef 00000080:00010000:2.0:1787695649.349807:0:98820:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.349811:0:98820:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bef from it ffffb5d404093c20 00010000:00010000:2.0:1787695649.349814:0:98820:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.349823:0:98820:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328bef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6b4 expref: -99 pid: 98820 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.350166:0:98820:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.350195:0:98820:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695649.350200:0:98820:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695649.350222:0:98820:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695649.350267:0:98820:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bf6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98820 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.350275:0:98820:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bf6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98820 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.350283:0:98820:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328bf6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98820 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.350305:0:98820:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98820:1874531134008960:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.350429:0:98820:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.386679:0:98821:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbd500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98821:1874531134009344:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.386710:0:98821:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.386733:0:98821:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.386745:0:98821:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.386756:0:98821:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.386769:0:98821:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.386777:0:98821:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.386784:0:98821:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.386797:0:98821:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babbd500 x1874531134009344/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.386812:0:98821:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.386825:0:98821:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.386833:0:98821:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.386903:0:98821:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.386911:0:98821:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328bfd 00000080:00010000:2.0:1787695649.386917:0:98821:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.386920:0:98821:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bfd from it ffffb5d403fe3be0 00010000:00010000:2.0:1787695649.386924:0:98821:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.387187:0:98821:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.387216:0:98821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.387242:0:98821:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.387247:0:98821:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.387266:0:98821:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.387276:0:98821:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.387288:0:98821:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.387295:0:98821:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.387300:0:98821:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.387303:0:98821:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328bfd 00000080:00010000:2.0:1787695649.387306:0:98821:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bfd from it ffffb5d403fe3ce0 00010000:00010000:2.0:1787695649.387310:0:98821:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.387317:0:98821:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.387331:0:98821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.387338:0:98821:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.387344:0:98821:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.387351:0:98821:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.387359:0:98821:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.387365:0:98821:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.387369:0:98821:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.387372:0:98821:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328bfd 00000080:00010000:2.0:1787695649.387375:0:98821:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328bfd from it ffffb5d403fe3dc8 00010000:00010000:2.0:1787695649.387378:0:98821:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.387384:0:98821:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.387391:0:98821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.387396:0:98821:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.387430:0:98821:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.387495:0:98821:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.387508:0:98821:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.387512:0:98821:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030c00/0x678ae32e95328bfd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed6f3 expref: -99 pid: 98821 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.387544:0:98821:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.387552:0:98821:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab77800 x1874531134009600/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.387577:0:98821:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab77800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98821:1874531134009600:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.387684:0:98821:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.394452:0:98821:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab77800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98821:1874531134009600:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.394527:0:98821:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x50c:0x0](ffff9d85b3cdc2c8) 00000080:00200000:2.0:1787695649.394537:0:98821:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x50c:0x0], updating i_size 512 00000080:00200000:2.0:1787695649.405614:0:98822:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.405644:0:98822:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.405662:0:98822:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695649.405796:0:98822:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98822 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.405808:0:98822:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98822 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.405819:0:98822:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98822 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.405921:0:98822:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98822:1874531134009728:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.406174:0:98822:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.410605:0:98822:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98822:1874531134009728:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.410653:0:98822:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410667:0:98822:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410681:0:98822:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410693:0:98822:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410708:0:98822:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410716:0:98822:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410724:0:98822:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.410747:0:98822:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab76d80 x1874531134009728/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.410761:0:98822:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410778:0:98822:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.410786:0:98822:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.410824:0:98822:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.410830:0:98822:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c04 00000080:00010000:2.0:1787695649.410836:0:98822:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.410840:0:98822:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c04 from it ffffb5d40402bc20 00010000:00010000:2.0:1787695649.410843:0:98822:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.410876:0:98822:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.410915:0:98822:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.410954:0:98822:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695649.410959:0:98822:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695649.410981:0:98822:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695649.411274:0:98822:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98822 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.411283:0:98822:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98822 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.411293:0:98822:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98822 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.411316:0:98822:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98822:1874531134009856:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.411461:0:98822:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.419699:0:98822:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98822:1874531134009856:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.419803:0:98822:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.419831:0:98822:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.419839:0:98822:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.420144:0:98822:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.420159:0:98822:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.420173:0:98822:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.420180:0:98822:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.420187:0:98822:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.420199:0:98822:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab76d80 x1874531134009856/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.420212:0:98822:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.420227:0:98822:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1292 (0 0) 00000080:00200000:2.0:1787695649.420234:0:98822:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695649.420270:0:98822:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x50c:0x0](ffff9d85b3cdaa08) 00000080:00200000:2.0:1787695649.420279:0:98822:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x50c:0x0], updating i_size 512 00000080:00010000:2.0:1787695649.420285:0:98822:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x50c:0x0](ffff9d85b3cdaa08) for lock 0x678ae32e95328c0b 00000080:00010000:2.0:1787695649.420295:0:98822:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x50c:0x0](ffff9d85b3cdaa08) 00000080:00010000:2.0:1787695649.420298:0:98822:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c0b from it ffffb5d40402be08 00010000:00010000:2.0:1787695649.420301:0:98822:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.420309:0:98822:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.420345:0:98822:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.420362:0:98822:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.420371:0:98822:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.420383:0:98822:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.420389:0:98822:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.420394:0:98822:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.420397:0:98822:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c04 00000080:00010000:2.0:1787695649.420401:0:98822:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c04 from it ffffb5d40402bd90 00010000:00010000:2.0:1787695649.420404:0:98822:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.420410:0:98822:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.420418:0:98822:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.420426:0:98822:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695649.420448:0:98822:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.420498:0:98822:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.420536:0:98822:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 1/0,0 mode: --/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.420543:0:98822:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.420549:0:98822:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.420551:0:98822:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c0b lrc: 0/0,0 mode: --/PR res: [0x200000401:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed72b expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.420569:0:98822:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c04 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed70f expref: -99 pid: 98822 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.420586:0:98822:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.420594:0:98822:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab76d80 x1874531134010112/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.420696:0:98822:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98822:1874531134010112:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.421139:0:98822:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.430279:0:98822:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98822:1874531134010112:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.430370:0:98822:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x50c:0x0](ffff9d85b3cdaa08) 00000080:00200000:2.0:1787695649.440841:0:98823:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.441096:0:98823:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.441161:0:98823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98823 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.441172:0:98823:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98823 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.441184:0:98823:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98823 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.441211:0:98823:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab77100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98823:1874531134010240:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695649.441363:0:98823:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.447200:0:98823:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab77100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98823:1874531134010240:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.447230:0:98823:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447244:0:98823:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447256:0:98823:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447273:0:98823:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447287:0:98823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447295:0:98823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447303:0:98823:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.447315:0:98823:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab77100 x1874531134010240/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.447330:0:98823:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447343:0:98823:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.447351:0:98823:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.447384:0:98823:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.447390:0:98823:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c12 00000080:00010000:2.0:1787695649.447396:0:98823:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.447399:0:98823:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c12 from it ffffb5d404043be0 00010000:00010000:2.0:1787695649.447403:0:98823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.447417:0:98823:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.447443:0:98823:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.447465:0:98823:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.447470:0:98823:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.448376:0:98823:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.448392:0:98823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.448406:0:98823:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.448412:0:98823:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.448419:0:98823:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.448422:0:98823:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c12 00000080:00010000:2.0:1787695649.448427:0:98823:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c12 from it ffffb5d404043ce0 00010000:00010000:2.0:1787695649.448431:0:98823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.448439:0:98823:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.448449:0:98823:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.448458:0:98823:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.448467:0:98823:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.448473:0:98823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.448491:0:98823:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.448497:0:98823:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.448501:0:98823:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.448503:0:98823:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c12 00000080:00010000:2.0:1787695649.448506:0:98823:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c12 from it ffffb5d404043dc8 00010000:00010000:2.0:1787695649.448508:0:98823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.448514:0:98823:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.448522:0:98823:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.448527:0:98823:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.448562:0:98823:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.448612:0:98823:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.448619:0:98823:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.448622:0:98823:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c12 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed74e expref: -99 pid: 98823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.448647:0:98823:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.448657:0:98823:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab77100 x1874531134010496/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.448686:0:98823:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab77100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98823:1874531134010496:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.448795:0:98823:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.456953:0:98823:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab77100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98823:1874531134010496:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.457017:0:98823:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x50d:0x0](ffff9d85b3cdaa08) 00000080:00200000:2.0:1787695649.457025:0:98823:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x50d:0x0], updating i_size 512 00000100:00100000:2.0:1787695649.469290:0:98824:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85ae913100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98824:1874531134010624:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.469423:0:98824:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.474132:0:98824:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85ae913100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98824:1874531134010624:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.474167:0:98824:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474180:0:98824:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474194:0:98824:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474205:0:98824:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474218:0:98824:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474226:0:98824:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474235:0:98824:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.474247:0:98824:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85ae913100 x1874531134010624/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.474260:0:98824:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474274:0:98824:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.474280:0:98824:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.474312:0:98824:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.474319:0:98824:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c19 00000080:00010000:2.0:1787695649.474323:0:98824:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.474326:0:98824:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c19 from it ffffb5d40409bc20 00010000:00010000:2.0:1787695649.474329:0:98824:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474338:0:98824:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.474363:0:98824:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.474388:0:98824:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695649.474394:0:98824:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695649.474418:0:98824:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695649.474470:0:98824:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98824 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474488:0:98824:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98824 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.474498:0:98824:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98824 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.474520:0:98824:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98824:1874531134010752:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.474640:0:98824:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.482443:0:98824:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98824:1874531134010752:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.482472:0:98824:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482504:0:98824:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482513:0:98824:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482524:0:98824:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482534:0:98824:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482545:0:98824:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482558:0:98824:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482565:0:98824:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.482577:0:98824:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab76680 x1874531134010752/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.482588:0:98824:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482598:0:98824:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1293 (0 0) 00000080:00200000:2.0:1787695649.482604:0:98824:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695649.482704:0:98824:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x50d:0x0](ffff9d85b3cde3c8) 00000080:00200000:2.0:1787695649.482714:0:98824:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x50d:0x0], updating i_size 512 00000080:00010000:2.0:1787695649.482722:0:98824:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x50d:0x0](ffff9d85b3cde3c8) for lock 0x678ae32e95328c20 00000080:00010000:2.0:1787695649.482802:0:98824:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x50d:0x0](ffff9d85b3cde3c8) 00000080:00010000:2.0:1787695649.482806:0:98824:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c20 from it ffffb5d40409be08 00010000:00010000:2.0:1787695649.482809:0:98824:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.482818:0:98824:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.483146:0:98824:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.483166:0:98824:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.483176:0:98824:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.483188:0:98824:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.483194:0:98824:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.483200:0:98824:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.483203:0:98824:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c19 00000080:00010000:2.0:1787695649.483207:0:98824:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c19 from it ffffb5d40409bd90 00010000:00010000:2.0:1787695649.483211:0:98824:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.483219:0:98824:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.483228:0:98824:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.483235:0:98824:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695649.483255:0:98824:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.483291:0:98824:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.483324:0:98824:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.483330:0:98824:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.483340:0:98824:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.483343:0:98824:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c20 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x50d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed786 expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.483359:0:98824:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328c19 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed76a expref: -99 pid: 98824 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.483375:0:98824:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.483382:0:98824:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab76680 x1874531134011008/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.483404:0:98824:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98824:1874531134011008:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.483531:0:98824:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.489720:0:98824:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98824:1874531134011008:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.489786:0:98824:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x50d:0x0](ffff9d85b3cde3c8) 00000080:00200000:2.0:1787695649.502703:0:98825:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.502803:0:98825:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.503182:0:98825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c27 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.503194:0:98825:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c27 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.503205:0:98825:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c27 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98825 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.503232:0:98825:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98825:1874531134011136:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695649.503383:0:98825:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.535489:0:98826:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98826:1874531134011520:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.535519:0:98826:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535530:0:98826:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535542:0:98826:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535553:0:98826:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535564:0:98826:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535572:0:98826:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535579:0:98826:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.535591:0:98826:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3b100 x1874531134011520/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.535605:0:98826:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535690:0:98826:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.535699:0:98826:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.535801:0:98826:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.535810:0:98826:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c2e 00000080:00010000:2.0:1787695649.535815:0:98826:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.535818:0:98826:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c2e from it ffffb5d403fe3c20 00010000:00010000:2.0:1787695649.535822:0:98826:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.535832:0:98826:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.536199:0:98826:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.536225:0:98826:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695649.536229:0:98826:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695649.536250:0:98826:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695649.536300:0:98826:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98826 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.536308:0:98826:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98826 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.536317:0:98826:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98826 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.536337:0:98826:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab77b80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98826:1874531134011648:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.536417:0:98826:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.539995:0:98826:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab77b80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98826:1874531134011648:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.540033:0:98826:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540054:0:98826:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540062:0:98826:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540073:0:98826:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540083:0:98826:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540096:0:98826:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540104:0:98826:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540111:0:98826:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.540123:0:98826:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab77b80 x1874531134011648/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.540136:0:98826:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540147:0:98826:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1294 (0 0) 00000080:00200000:2.0:1787695649.540154:0:98826:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695649.540188:0:98826:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x50e:0x0](ffff9d85b3cde3c8) 00000080:00200000:2.0:1787695649.540196:0:98826:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x50e:0x0], updating i_size 512 00000080:00010000:2.0:1787695649.540203:0:98826:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x50e:0x0](ffff9d85b3cde3c8) for lock 0x678ae32e95328c35 00000080:00010000:2.0:1787695649.540215:0:98826:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x50e:0x0](ffff9d85b3cde3c8) 00000080:00010000:2.0:1787695649.540218:0:98826:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c35 from it ffffb5d403fe3e08 00010000:00010000:2.0:1787695649.540221:0:98826:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.540230:0:98826:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.540260:0:98826:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.540275:0:98826:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.540284:0:98826:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.540293:0:98826:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.540299:0:98826:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.540304:0:98826:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.540307:0:98826:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c2e 00000080:00010000:2.0:1787695649.540310:0:98826:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c2e from it ffffb5d403fe3d90 00010000:00010000:2.0:1787695649.540313:0:98826:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.540320:0:98826:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.540328:0:98826:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.540335:0:98826:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695649.540354:0:98826:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.540388:0:98826:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.540422:0:98826:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.540429:0:98826:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.540435:0:98826:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.540438:0:98826:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c35 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x50e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed7e1 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.540454:0:98826:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328c2e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed7c5 expref: -99 pid: 98826 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.540470:0:98826:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.540489:0:98826:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab77b80 x1874531134011904/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.540517:0:98826:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab77b80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98826:1874531134011904:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.540587:0:98826:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.546337:0:98826:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab77b80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98826:1874531134011904:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.546381:0:98826:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x50e:0x0](ffff9d85b3cde3c8) 00000080:00200000:2.0:1787695649.691358:0:98831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.691383:0:98831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.691450:0:98831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98831 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.691461:0:98831:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98831 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.691470:0:98831:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98831 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.691508:0:98831:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76a00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98831:1874531134013824:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695649.691673:0:98831:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.697518:0:98831:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76a00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98831:1874531134013824:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.697546:0:98831:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697558:0:98831:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697569:0:98831:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697585:0:98831:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697598:0:98831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697607:0:98831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697614:0:98831:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.697638:0:98831:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab76a00 x1874531134013824/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.697653:0:98831:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697669:0:98831:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.697677:0:98831:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.697709:0:98831:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.697716:0:98831:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c66 00000080:00010000:2.0:1787695649.697722:0:98831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.697725:0:98831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c66 from it ffffb5d40402bbe0 00010000:00010000:2.0:1787695649.697729:0:98831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.697742:0:98831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.697770:0:98831:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.697796:0:98831:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.697801:0:98831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.697817:0:98831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.697828:0:98831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.697839:0:98831:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.697845:0:98831:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.698111:0:98831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.698116:0:98831:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c66 00000080:00010000:2.0:1787695649.698121:0:98831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c66 from it ffffb5d40402bce0 00010000:00010000:2.0:1787695649.698124:0:98831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.698133:0:98831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.698142:0:98831:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.698150:0:98831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.698158:0:98831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.698165:0:98831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.698173:0:98831:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.698180:0:98831:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.698184:0:98831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.698186:0:98831:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c66 00000080:00010000:2.0:1787695649.698190:0:98831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c66 from it ffffb5d40402bdc8 00010000:00010000:2.0:1787695649.698194:0:98831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.698201:0:98831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.698209:0:98831:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.698214:0:98831:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.698245:0:98831:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.698288:0:98831:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.698296:0:98831:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.698298:0:98831:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c66 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed8ba expref: -99 pid: 98831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.698322:0:98831:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.698330:0:98831:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab76a00 x1874531134014080/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.698351:0:98831:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76a00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98831:1874531134014080:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.698444:0:98831:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.704026:0:98831:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76a00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98831:1874531134014080:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.704079:0:98831:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x511:0x0](ffff9d85b3cde3c8) 00000080:00200000:2.0:1787695649.704087:0:98831:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x511:0x0], updating i_size 512 00000080:00200000:2.0:1787695649.716287:0:98832:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.716317:0:98832:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.716333:0:98832:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695649.716390:0:98832:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98832 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.716401:0:98832:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98832 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.716413:0:98832:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98832 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.716444:0:98832:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98832:1874531134014208:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.716549:0:98832:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.720436:0:98832:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98832:1874531134014208:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.720465:0:98832:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.720488:0:98832:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.720504:0:98832:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.720515:0:98832:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.720528:0:98832:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.720535:0:98832:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.720542:0:98832:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.720555:0:98832:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab74700 x1874531134014208/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.720568:0:98832:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.720580:0:98832:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.720586:0:98832:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.720615:0:98832:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.720621:0:98832:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c6d 00000080:00010000:2.0:1787695649.721096:0:98832:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.721101:0:98832:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c6d from it ffffb5d404093c20 00010000:00010000:2.0:1787695649.721104:0:98832:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.721114:0:98832:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.721139:0:98832:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.721162:0:98832:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695649.721166:0:98832:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695649.721184:0:98832:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695649.721228:0:98832:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98832 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.721235:0:98832:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98832 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.721243:0:98832:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98832 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.721262:0:98832:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98832:1874531134014336:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695649.721398:0:98832:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.726407:0:98832:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98832:1874531134014336:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695649.726436:0:98832:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726456:0:98832:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x511:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726465:0:98832:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726485:0:98832:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726497:0:98832:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726510:0:98832:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726518:0:98832:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726525:0:98832:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.726536:0:98832:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab74700 x1874531134014336/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.726548:0:98832:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726558:0:98832:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1297 (0 0) 00000080:00200000:2.0:1787695649.726566:0:98832:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695649.726603:0:98832:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x511:0x0](ffff9d85b3cda1c8) 00000080:00200000:2.0:1787695649.726611:0:98832:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x511:0x0], updating i_size 512 00000080:00010000:2.0:1787695649.726618:0:98832:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x511:0x0](ffff9d85b3cda1c8) for lock 0x678ae32e95328c74 00000080:00010000:2.0:1787695649.726641:0:98832:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x511:0x0](ffff9d85b3cda1c8) 00000080:00010000:2.0:1787695649.726646:0:98832:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c74 from it ffffb5d404093e08 00010000:00010000:2.0:1787695649.726649:0:98832:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.726657:0:98832:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.726687:0:98832:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695649.726704:0:98832:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.726713:0:98832:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.726723:0:98832:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.726730:0:98832:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.726734:0:98832:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695649.726737:0:98832:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328c6d 00000080:00010000:2.0:1787695649.726741:0:98832:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c6d from it ffffb5d404093d90 00010000:00010000:2.0:1787695649.726743:0:98832:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.726750:0:98832:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.726759:0:98832:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.726766:0:98832:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695649.726784:0:98832:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.726816:0:98832:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.726882:0:98832:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.726900:0:98832:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.726906:0:98832:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.726909:0:98832:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328c74 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x511:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed8f2 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.726924:0:98832:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328c6d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed8d6 expref: -99 pid: 98832 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.726947:0:98832:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.726953:0:98832:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab74700 x1874531134014592/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.726974:0:98832:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98832:1874531134014592:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.727037:0:98832:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.732568:0:98832:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98832:1874531134014592:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.732612:0:98832:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x511:0x0](ffff9d85b3cda1c8) 00000080:00200000:2.0:1787695649.752515:0:98833:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.752542:0:98833:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.752608:0:98833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98833 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.752619:0:98833:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98833 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.752798:0:98833:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98833 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.752827:0:98833:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98833:1874531134014720:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695649.753229:0:98833:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.759297:0:98833:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98833:1874531134014720:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.759323:0:98833:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.759336:0:98833:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.759348:0:98833:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.759359:0:98833:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.759372:0:98833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.759380:0:98833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.759392:0:98833:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.759403:0:98833:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab76300 x1874531134014720/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.759417:0:98833:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328c7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed915 expref: -99 pid: 98833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.759429:0:98833:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.759435:0:98833:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00010000:00010000:2.0:1787695649.795596:0:98834:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328c89 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x512:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed94d expref: -99 pid: 98834 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.795605:0:98834:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328c82 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed931 expref: -99 pid: 98834 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.795611:0:98834:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695649.795613:0:98834:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328c89 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x512:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed94d expref: -99 pid: 98834 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.795631:0:98834:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328c82 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed931 expref: -99 pid: 98834 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695649.795646:0:98834:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.795655:0:98834:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab75c00 x1874531134015488/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.795684:0:98834:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98834:1874531134015488:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695649.795768:0:98834:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.800001:0:98834:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98834:1874531134015488:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695649.800045:0:98834:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x512:0x0](ffff9d85b3cc9148) 00000080:00200000:2.0:1787695649.807685:0:98835:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.807716:0:98835:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.807791:0:98835:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98835 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.807804:0:98835:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98835 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.807817:0:98835:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98835 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.808154:0:98835:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98835:1874531134015616:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695649.808261:0:98835:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.815413:0:98835:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98835:1874531134015616:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.815444:0:98835:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815457:0:98835:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815469:0:98835:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815497:0:98835:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815512:0:98835:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815520:0:98835:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815528:0:98835:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.815541:0:98835:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab74e00 x1874531134015616/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.815557:0:98835:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815571:0:98835:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.815579:0:98835:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.815608:0:98835:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.815614:0:98835:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c90 00000080:00010000:2.0:1787695649.815620:0:98835:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.815623:0:98835:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c90 from it ffffb5d403fe3be0 00010000:00010000:2.0:1787695649.815627:0:98835:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.815641:0:98835:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.815669:0:98835:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.815695:0:98835:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.815700:0:98835:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.815717:0:98835:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.815727:0:98835:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.815738:0:98835:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.815744:0:98835:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.815749:0:98835:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.815751:0:98835:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c90 00000080:00010000:2.0:1787695649.815755:0:98835:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c90 from it ffffb5d403fe3ce0 00010000:00010000:2.0:1787695649.815758:0:98835:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.815765:0:98835:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.815773:0:98835:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.815780:0:98835:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.815786:0:98835:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.815793:0:98835:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.815800:0:98835:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.815806:0:98835:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.815810:0:98835:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.815812:0:98835:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328c90 00000080:00010000:2.0:1787695649.815816:0:98835:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328c90 from it ffffb5d403fe3dc8 00010000:00010000:2.0:1787695649.815819:0:98835:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.815842:0:98835:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.815880:0:98835:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.815886:0:98835:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.815926:0:98835:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.815967:0:98835:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.815974:0:98835:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.815977:0:98835:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328c90 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed970 expref: -99 pid: 98835 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.816002:0:98835:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.816009:0:98835:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab74e00 x1874531134015872/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.816030:0:98835:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98835:1874531134015872:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.816267:0:98835:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.822778:0:98835:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98835:1874531134015872:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.822828:0:98835:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x513:0x0](ffff9d85b3cda1c8) 00000080:00200000:2.0:1787695649.822835:0:98835:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x513:0x0], updating i_size 512 00000100:00100000:2.0:1787695649.898530:0:98837:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98837:1874531134016512:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.898556:0:98837:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898569:0:98837:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898579:0:98837:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898588:0:98837:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898600:0:98837:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898607:0:98837:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898614:0:98837:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.898639:0:98837:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f38e00 x1874531134016512/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.898652:0:98837:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898665:0:98837:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.898671:0:98837:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.898697:0:98837:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.898703:0:98837:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ca5 00000080:00010000:2.0:1787695649.898708:0:98837:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.898711:0:98837:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ca5 from it ffffb5d4040a3be0 00010000:00010000:2.0:1787695649.898714:0:98837:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.898722:0:98837:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.898746:0:98837:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.898766:0:98837:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.898770:0:98837:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.898782:0:98837:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.898791:0:98837:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.898801:0:98837:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.898812:0:98837:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.898816:0:98837:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.898819:0:98837:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ca5 00000080:00010000:2.0:1787695649.898822:0:98837:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ca5 from it ffffb5d4040a3ce0 00010000:00010000:2.0:1787695649.898825:0:98837:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.898833:0:98837:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.898841:0:98837:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.898847:0:98837:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.898884:0:98837:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.898890:0:98837:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.898899:0:98837:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.898906:0:98837:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.898910:0:98837:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.898912:0:98837:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ca5 00000080:00010000:2.0:1787695649.898915:0:98837:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ca5 from it ffffb5d4040a3dc8 00010000:00010000:2.0:1787695649.898918:0:98837:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.898940:0:98837:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.898949:0:98837:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.898955:0:98837:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.898987:0:98837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.899046:0:98837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.899054:0:98837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.899057:0:98837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c400/0x678ae32e95328ca5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ed9cb expref: -99 pid: 98837 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.899081:0:98837:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.899088:0:98837:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab77480 x1874531134016768/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.899111:0:98837:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab77480 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98837:1874531134016768:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.899189:0:98837:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000080:00200000:2.0:1787695649.949708:0:98839:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.949737:0:98839:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.949803:0:98839:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98839 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.949814:0:98839:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98839 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.949825:0:98839:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98839 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695649.949875:0:98839:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98839:1874531134017408:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695649.950482:0:98839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.953832:0:98839:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98839:1874531134017408:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.954093:0:98839:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954107:0:98839:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954118:0:98839:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954131:0:98839:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954144:0:98839:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954153:0:98839:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954160:0:98839:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.954178:0:98839:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab75f80 x1874531134017408/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.954194:0:98839:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954208:0:98839:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.954215:0:98839:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.954250:0:98839:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.954257:0:98839:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328cba 00000080:00010000:2.0:1787695649.954262:0:98839:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.954265:0:98839:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328cba from it ffffb5d4040c3be0 00010000:00010000:2.0:1787695649.954268:0:98839:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.954276:0:98839:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.954303:0:98839:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.954325:0:98839:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.954330:0:98839:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.954343:0:98839:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.954353:0:98839:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.954363:0:98839:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.954373:0:98839:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.954377:0:98839:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.954381:0:98839:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328cba 00000080:00010000:2.0:1787695649.954384:0:98839:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328cba from it ffffb5d4040c3ce0 00010000:00010000:2.0:1787695649.954387:0:98839:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.954394:0:98839:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.954402:0:98839:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.954408:0:98839:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.954414:0:98839:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.954420:0:98839:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.954427:0:98839:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.954433:0:98839:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.954436:0:98839:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.954438:0:98839:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328cba 00000080:00010000:2.0:1787695649.954441:0:98839:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328cba from it ffffb5d4040c3dc8 00010000:00010000:2.0:1787695649.954444:0:98839:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.954455:0:98839:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.954462:0:98839:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.954466:0:98839:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.954509:0:98839:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.954555:0:98839:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.954688:0:98839:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.954692:0:98839:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328cba lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eda26 expref: -99 pid: 98839 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.954721:0:98839:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.954729:0:98839:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab75f80 x1874531134017664/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.954751:0:98839:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98839:1874531134017664:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.954894:0:98839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695649.960899:0:98839:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98839:1874531134017664:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695649.961082:0:98839:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x515:0x0](ffff9d85b3cd80c8) 00000080:00200000:2.0:1787695649.961092:0:98839:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x515:0x0], updating i_size 512 00000100:00100000:2.0:1787695649.993976:0:98841:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbca80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98841:1874531134018304:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695649.994009:0:98841:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994023:0:98841:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994034:0:98841:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994045:0:98841:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994056:0:98841:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994065:0:98841:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994072:0:98841:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695649.994083:0:98841:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babbca80 x1874531134018304/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695666 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695649.994097:0:98841:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994107:0:98841:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695649.994114:0:98841:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695649.994141:0:98841:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695649.994147:0:98841:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ccf 00000080:00010000:2.0:1787695649.994152:0:98841:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.994155:0:98841:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ccf from it ffffb5d4040cbbe0 00010000:00010000:2.0:1787695649.994159:0:98841:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695649.994167:0:98841:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.994189:0:98841:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695649.994208:0:98841:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695649.994212:0:98841:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.994225:0:98841:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.994232:0:98841:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.994241:0:98841:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695649.994248:0:98841:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.994252:0:98841:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.994254:0:98841:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ccf 00000080:00010000:2.0:1787695649.994258:0:98841:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ccf from it ffffb5d4040cbce0 00010000:00010000:2.0:1787695649.994261:0:98841:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695649.994268:0:98841:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.994282:0:98841:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.994287:0:98841:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695649.994292:0:98841:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695649.994297:0:98841:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.994304:0:98841:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695649.994310:0:98841:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695649.994313:0:98841:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695649.994315:0:98841:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ccf 00000080:00010000:2.0:1787695649.994318:0:98841:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ccf from it ffffb5d4040cbdc8 00010000:00010000:2.0:1787695649.994321:0:98841:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695649.994328:0:98841:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695649.994336:0:98841:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695649.994340:0:98841:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695649.994364:0:98841:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.994399:0:98841:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695649.994411:0:98841:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695649.994414:0:98841:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030e00/0x678ae32e95328ccf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eda81 expref: -99 pid: 98841 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695649.994435:0:98841:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695649.994441:0:98841:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab74000 x1874531134018560/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695649.994460:0:98841:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98841:1874531134018560:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695649.994545:0:98841:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.000055:0:98841:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98841:1874531134018560:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695650.000100:0:98841:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x516:0x0](ffff9d85b3cd8908) 00000080:00200000:2.0:1787695650.000108:0:98841:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x516:0x0], updating i_size 512 00000080:00200000:2.0:1787695650.007793:0:98842:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.007826:0:98842:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.007840:0:98842:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695650.010411:0:98842:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98842 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.010431:0:98842:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98842 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.010441:0:98842:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98842 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.010503:0:98842:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98842:1874531134018688:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.010641:0:98842:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.014897:0:98842:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98842:1874531134018688:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.014931:0:98842:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.014944:0:98842:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.014957:0:98842:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.014968:0:98842:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.014981:0:98842:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.014989:0:98842:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.014996:0:98842:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.015008:0:98842:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab74380 x1874531134018688/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.015021:0:98842:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.015033:0:98842:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.015040:0:98842:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.015069:0:98842:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.015075:0:98842:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328cd6 00000080:00010000:2.0:1787695650.015079:0:98842:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.015082:0:98842:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328cd6 from it ffffb5d4040d3c20 00010000:00010000:2.0:1787695650.015085:0:98842:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.015093:0:98842:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.015121:0:98842:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.015146:0:98842:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.015151:0:98842:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.015172:0:98842:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.015217:0:98842:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98842 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.015225:0:98842:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98842 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.015234:0:98842:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98842 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.015255:0:98842:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98842:1874531134018816:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.015323:0:98842:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.018493:0:98842:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98842:1874531134018816:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.018521:0:98842:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018543:0:98842:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 4/1,0 mode: --/PR res: [0x200000401:0x516:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018551:0:98842:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 4/1,0 mode: --/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018561:0:98842:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018572:0:98842:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018584:0:98842:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018592:0:98842:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018599:0:98842:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.018611:0:98842:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab74380 x1874531134018816/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.018635:0:98842:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018651:0:98842:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1302 (0 0) 00000080:00200000:2.0:1787695650.018658:0:98842:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695650.018688:0:98842:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x516:0x0](ffff9d85b3cd9148) 00000080:00200000:2.0:1787695650.018695:0:98842:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x516:0x0], updating i_size 512 00000080:00010000:2.0:1787695650.018701:0:98842:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x516:0x0](ffff9d85b3cd9148) for lock 0x678ae32e95328cdd 00000080:00010000:2.0:1787695650.018710:0:98842:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x516:0x0](ffff9d85b3cd9148) 00000080:00010000:2.0:1787695650.018713:0:98842:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328cdd from it ffffb5d4040d3e08 00010000:00010000:2.0:1787695650.018716:0:98842:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.018724:0:98842:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.018753:0:98842:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.018769:0:98842:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.018778:0:98842:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.018788:0:98842:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.018794:0:98842:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.018798:0:98842:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.018800:0:98842:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328cd6 00000080:00010000:2.0:1787695650.018804:0:98842:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328cd6 from it ffffb5d4040d3d90 00010000:00010000:2.0:1787695650.018807:0:98842:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.018815:0:98842:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.018824:0:98842:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.018831:0:98842:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695650.018888:0:98842:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.018922:0:98842:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.019158:0:98842:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 1/0,0 mode: --/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.019167:0:98842:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.019172:0:98842:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695650.019175:0:98842:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328cdd lrc: 0/0,0 mode: --/PR res: [0x200000401:0x516:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edab9 expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.019192:0:98842:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cd6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eda9d expref: -99 pid: 98842 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695650.019206:0:98842:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.019213:0:98842:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab74380 x1874531134019072/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.019237:0:98842:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98842:1874531134019072:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695650.019347:0:98842:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.025047:0:98842:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98842:1874531134019072:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695650.025441:0:98842:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x516:0x0](ffff9d85b3cd9148) 00000100:00100000:2.0:1787695650.057434:0:98844:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbed80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98844:1874531134019584:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.057460:0:98844:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057483:0:98844:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057495:0:98844:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057506:0:98844:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057519:0:98844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057527:0:98844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057534:0:98844:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.057550:0:98844:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babbed80 x1874531134019584/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.057563:0:98844:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057575:0:98844:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.057582:0:98844:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.057608:0:98844:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.057614:0:98844:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ceb 00000080:00010000:2.0:1787695650.057618:0:98844:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.057621:0:98844:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ceb from it ffffb5d4040dbc20 00010000:00010000:2.0:1787695650.057686:0:98844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057696:0:98844:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.057720:0:98844:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.057747:0:98844:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.057751:0:98844:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.057769:0:98844:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.057809:0:98844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057816:0:98844:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.057830:0:98844:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98844 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.058080:0:98844:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98844:1874531134019712:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.058181:0:98844:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.061592:0:98844:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98844:1874531134019712:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.061620:0:98844:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061739:0:98844:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x517:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061747:0:98844:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061758:0:98844:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061769:0:98844:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061784:0:98844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061792:0:98844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061803:0:98844:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.061814:0:98844:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab75180 x1874531134019712/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.061827:0:98844:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.061837:0:98844:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1303 (0 0) 00000080:00200000:2.0:1787695650.061843:0:98844:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695650.062253:0:98844:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x517:0x0](ffff9d85b3cd9148) 00000080:00200000:2.0:1787695650.062263:0:98844:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x517:0x0], updating i_size 512 00000080:00010000:2.0:1787695650.062271:0:98844:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x517:0x0](ffff9d85b3cd9148) for lock 0x678ae32e95328cf2 00000080:00010000:2.0:1787695650.062282:0:98844:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x517:0x0](ffff9d85b3cd9148) 00000080:00010000:2.0:1787695650.062285:0:98844:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328cf2 from it ffffb5d4040dbe08 00010000:00010000:2.0:1787695650.062289:0:98844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.062298:0:98844:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.062328:0:98844:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.062344:0:98844:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.062353:0:98844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.062369:0:98844:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.062375:0:98844:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.062381:0:98844:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.062384:0:98844:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ceb 00000080:00010000:2.0:1787695650.062388:0:98844:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ceb from it ffffb5d4040dbd90 00010000:00010000:2.0:1787695650.062390:0:98844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.062398:0:98844:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.062406:0:98844:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.062415:0:98844:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695650.062435:0:98844:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.062485:0:98844:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.062524:0:98844:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.062531:0:98844:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.062542:0:98844:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695650.062545:0:98844:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328cf2 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x517:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edb14 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.062560:0:98844:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e95328ceb lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edaf8 expref: -99 pid: 98844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695650.062575:0:98844:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.062583:0:98844:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab75180 x1874531134019968/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.062606:0:98844:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98844:1874531134019968:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695650.062790:0:98844:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.068369:0:98844:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98844:1874531134019968:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695650.068413:0:98844:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x517:0x0](ffff9d85b3cd9148) 00000100:00100000:2.0:1787695650.111154:0:98846:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbd500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98846:1874531134020480:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.111183:0:98846:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111197:0:98846:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111209:0:98846:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111221:0:98846:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111234:0:98846:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111242:0:98846:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111250:0:98846:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.111263:0:98846:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babbd500 x1874531134020480/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.111277:0:98846:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111291:0:98846:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.111309:0:98846:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.111340:0:98846:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.111347:0:98846:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328d00 00000080:00010000:2.0:1787695650.111353:0:98846:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.111356:0:98846:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d00 from it ffffb5d40402bc20 00010000:00010000:2.0:1787695650.111360:0:98846:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111370:0:98846:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.111405:0:98846:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.111437:0:98846:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.111442:0:98846:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.111502:0:98846:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.111559:0:98846:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98846 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111567:0:98846:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98846 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.111577:0:98846:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98846 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.111599:0:98846:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98846:1874531134020608:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.111671:0:98846:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.115797:0:98846:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98846:1874531134020608:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.115832:0:98846:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.115896:0:98846:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x518:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.115904:0:98846:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.115916:0:98846:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.116111:0:98846:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.116125:0:98846:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.116134:0:98846:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.116142:0:98846:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.116154:0:98846:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab74a80 x1874531134020608/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.116167:0:98846:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.116179:0:98846:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1304 (0 0) 00000080:00200000:2.0:1787695650.116187:0:98846:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695650.116220:0:98846:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x518:0x0](ffff9d85b3cd9148) 00000080:00200000:2.0:1787695650.116228:0:98846:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x518:0x0], updating i_size 512 00000080:00010000:2.0:1787695650.116236:0:98846:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x518:0x0](ffff9d85b3cd9148) for lock 0x678ae32e95328d07 00000080:00010000:2.0:1787695650.116246:0:98846:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x518:0x0](ffff9d85b3cd9148) 00000080:00010000:2.0:1787695650.116249:0:98846:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d07 from it ffffb5d40402be08 00010000:00010000:2.0:1787695650.116253:0:98846:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.116267:0:98846:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.116302:0:98846:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.116321:0:98846:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.116331:0:98846:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.116342:0:98846:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.116348:0:98846:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.116352:0:98846:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.116356:0:98846:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328d00 00000080:00010000:2.0:1787695650.116359:0:98846:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d00 from it ffffb5d40402bd90 00010000:00010000:2.0:1787695650.116362:0:98846:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.116369:0:98846:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.116378:0:98846:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.116385:0:98846:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695650.116404:0:98846:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.116437:0:98846:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.116483:0:98846:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.116491:0:98846:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.116496:0:98846:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695650.116499:0:98846:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d07 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x518:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edb6f expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.116515:0:98846:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031a00/0x678ae32e95328d00 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edb53 expref: -99 pid: 98846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695650.116532:0:98846:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.116539:0:98846:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab74a80 x1874531134020864/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.116564:0:98846:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab74a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98846:1874531134020864:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695650.116708:0:98846:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.121387:0:98846:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab74a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98846:1874531134020864:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695650.121433:0:98846:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x518:0x0](ffff9d85b3cd9148) 00000080:00200000:2.0:1787695650.259681:0:98852:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.259714:0:98852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.259729:0:98852:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695650.259786:0:98852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.259800:0:98852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.259809:0:98852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.259837:0:98852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98852:1874531134023168:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.259946:0:98852:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.265158:0:98852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98852:1874531134023168:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.265188:0:98852:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265201:0:98852:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265212:0:98852:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265223:0:98852:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265235:0:98852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265243:0:98852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265254:0:98852:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.265265:0:98852:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab76300 x1874531134023168/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.265279:0:98852:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265292:0:98852:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.265300:0:98852:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.265330:0:98852:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.265336:0:98852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328d3f 00000080:00010000:2.0:1787695650.265341:0:98852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.265344:0:98852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d3f from it ffffb5d4040b3c20 00010000:00010000:2.0:1787695650.265347:0:98852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265356:0:98852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.265383:0:98852:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.265521:0:98852:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.265527:0:98852:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.265550:0:98852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.265591:0:98852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265597:0:98852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.265606:0:98852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.265686:0:98852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98852:1874531134023296:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.266052:0:98852:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.269575:0:98852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98852:1874531134023296:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.269608:0:98852:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269643:0:98852:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269652:0:98852:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269664:0:98852:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269676:0:98852:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269689:0:98852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269703:0:98852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269709:0:98852:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.269721:0:98852:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab76300 x1874531134023296/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.269735:0:98852:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269745:0:98852:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1307 (0 0) 00000080:00200000:2.0:1787695650.269751:0:98852:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695650.269783:0:98852:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x51b:0x0](ffff9d85b3cd9148) 00000080:00200000:2.0:1787695650.269792:0:98852:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x51b:0x0], updating i_size 512 00000080:00010000:2.0:1787695650.269798:0:98852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x51b:0x0](ffff9d85b3cd9148) for lock 0x678ae32e95328d46 00000080:00010000:2.0:1787695650.269809:0:98852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x51b:0x0](ffff9d85b3cd9148) 00000080:00010000:2.0:1787695650.269813:0:98852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d46 from it ffffb5d4040b3e08 00010000:00010000:2.0:1787695650.269817:0:98852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.269826:0:98852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.269888:0:98852:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.269908:0:98852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.270092:0:98852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.270106:0:98852:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.270113:0:98852:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.270118:0:98852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.270122:0:98852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328d3f 00000080:00010000:2.0:1787695650.270126:0:98852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d3f from it ffffb5d4040b3d90 00010000:00010000:2.0:1787695650.270129:0:98852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.270136:0:98852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.270146:0:98852:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.270155:0:98852:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695650.270179:0:98852:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.270217:0:98852:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.270255:0:98852:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.270264:0:98852:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.270276:0:98852:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695650.270280:0:98852:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d46 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x51b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edc80 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.270299:0:98852:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d3f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edc64 expref: -99 pid: 98852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695650.270318:0:98852:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.270327:0:98852:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab76300 x1874531134023552/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.270350:0:98852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98852:1874531134023552:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695650.270428:0:98852:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.276433:0:98852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab76300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98852:1874531134023552:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695650.276503:0:98852:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x51b:0x0](ffff9d85b3cd9148) 00000080:00200000:2.0:1787695650.288233:0:98853:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.288261:0:98853:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.288326:0:98853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98853 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.288338:0:98853:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98853 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.288348:0:98853:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98853 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.288373:0:98853:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98853:1874531134023680:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695650.288484:0:98853:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.291841:0:98853:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98853:1874531134023680:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695650.291988:0:98853:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292100:0:98853:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292112:0:98853:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292122:0:98853:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292136:0:98853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292145:0:98853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292151:0:98853:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.292162:0:98853:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85bab75500 x1874531134023680/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.292176:0:98853:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292195:0:98853:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.292202:0:98853:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.292233:0:98853:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.292240:0:98853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328d4d 00000080:00010000:2.0:1787695650.292245:0:98853:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.292248:0:98853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d4d from it ffffb5d4040fbbe0 00010000:00010000:2.0:1787695650.292252:0:98853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.292261:0:98853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.292286:0:98853:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.292308:0:98853:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695650.292312:0:98853:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.292326:0:98853:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.292336:0:98853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.292346:0:98853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.292352:0:98853:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.292356:0:98853:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.292359:0:98853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328d4d 00000080:00010000:2.0:1787695650.292363:0:98853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d4d from it ffffb5d4040fbce0 00010000:00010000:2.0:1787695650.292366:0:98853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.292372:0:98853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.292380:0:98853:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.292386:0:98853:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.292391:0:98853:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.292397:0:98853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.292404:0:98853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.292409:0:98853:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.292412:0:98853:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.292415:0:98853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328d4d 00000080:00010000:2.0:1787695650.292418:0:98853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d4d from it ffffb5d4040fbdc8 00010000:00010000:2.0:1787695650.292421:0:98853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.292427:0:98853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695650.292434:0:98853:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.292439:0:98853:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695650.292497:0:98853:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.292548:0:98853:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.292555:0:98853:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695650.292558:0:98853:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328d4d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edca3 expref: -99 pid: 98853 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695650.292586:0:98853:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.292593:0:98853:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab75500 x1874531134023936/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.292615:0:98853:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98853:1874531134023936:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695650.333833:0:98855:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babbd500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98855:1874531134024576:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695650.333889:0:98855:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.333903:0:98855:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.333923:0:98855:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.333935:0:98855:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.333948:0:98855:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.333957:0:98855:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.333964:0:98855:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.333976:0:98855:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babbd500 x1874531134024576/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.334208:0:98855:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.334224:0:98855:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.334232:0:98855:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.334261:0:98855:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.334267:0:98855:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328d62 00000080:00010000:2.0:1787695650.334273:0:98855:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.334276:0:98855:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d62 from it ffffb5d404093be0 00010000:00010000:2.0:1787695650.334279:0:98855:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.334288:0:98855:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.334312:0:98855:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.334331:0:98855:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695650.334336:0:98855:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.334351:0:98855:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.334359:0:98855:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.334368:0:98855:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.334374:0:98855:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.334378:0:98855:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.334381:0:98855:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328d62 00000080:00010000:2.0:1787695650.334385:0:98855:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d62 from it ffffb5d404093ce0 00010000:00010000:2.0:1787695650.334388:0:98855:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.334395:0:98855:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.334404:0:98855:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.334410:0:98855:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.334416:0:98855:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.334422:0:98855:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.334429:0:98855:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.334436:0:98855:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.334439:0:98855:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.334441:0:98855:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328d62 00000080:00010000:2.0:1787695650.334445:0:98855:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d62 from it ffffb5d404093dc8 00010000:00010000:2.0:1787695650.334447:0:98855:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.334454:0:98855:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695650.334472:0:98855:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.334479:0:98855:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695650.334516:0:98855:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.334567:0:98855:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.334574:0:98855:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695650.334577:0:98855:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95328d62 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edcfe expref: -99 pid: 98855 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695650.334601:0:98855:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.334609:0:98855:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85bab75880 x1874531134024832/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.334641:0:98855:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85bab75880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98855:1874531134024832:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695650.334754:0:98855:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.341311:0:98855:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85bab75880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98855:1874531134024832:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695650.341365:0:98855:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x51d:0x0](ffff9d85b3cd9148) 00000080:00200000:2.0:1787695650.341372:0:98855:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x51d:0x0], updating i_size 512 00000080:00200000:2.0:1787695650.355676:0:98856:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.355707:0:98856:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.355722:0:98856:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695650.355794:0:98856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98856 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.355803:0:98856:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98856 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.355813:0:98856:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98856 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.355837:0:98856:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98856:1874531134024960:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.355948:0:98856:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.364494:0:98856:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98856:1874531134024960:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.364525:0:98856:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364537:0:98856:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364548:0:98856:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364559:0:98856:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364573:0:98856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364586:0:98856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364593:0:98856:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.364604:0:98856:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce3480 x1874531134024960/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.364617:0:98856:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364644:0:98856:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.364652:0:98856:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.364684:0:98856:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.364691:0:98856:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328d69 00000080:00010000:2.0:1787695650.364696:0:98856:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.364699:0:98856:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d69 from it ffffb5d404103c20 00010000:00010000:2.0:1787695650.364703:0:98856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364711:0:98856:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.364734:0:98856:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.364763:0:98856:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.364768:0:98856:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.364793:0:98856:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.364838:0:98856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98856 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364846:0:98856:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98856 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.364882:0:98856:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98856 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.364904:0:98856:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98856:1874531134025088:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.365342:0:98856:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.370535:0:98856:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98856:1874531134025088:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.370572:0:98856:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370595:0:98856:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370603:0:98856:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370615:0:98856:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370709:0:98856:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370725:0:98856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370734:0:98856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370740:0:98856:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.370752:0:98856:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce3480 x1874531134025088/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.370764:0:98856:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370774:0:98856:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1309 (0 0) 00000080:00200000:2.0:1787695650.370782:0:98856:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695650.370816:0:98856:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x51d:0x0](ffff9d85b3cdf448) 00000080:00200000:2.0:1787695650.370827:0:98856:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x51d:0x0], updating i_size 512 00000080:00010000:2.0:1787695650.370834:0:98856:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x51d:0x0](ffff9d85b3cdf448) for lock 0x678ae32e95328d70 00000080:00010000:2.0:1787695650.370844:0:98856:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x51d:0x0](ffff9d85b3cdf448) 00000080:00010000:2.0:1787695650.370848:0:98856:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d70 from it ffffb5d404103e08 00010000:00010000:2.0:1787695650.370879:0:98856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.370888:0:98856:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.371183:0:98856:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.371212:0:98856:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.371224:0:98856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.371238:0:98856:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.371245:0:98856:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.371250:0:98856:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.371253:0:98856:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328d69 00000080:00010000:2.0:1787695650.371258:0:98856:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328d69 from it ffffb5d404103d90 00010000:00010000:2.0:1787695650.371261:0:98856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.371267:0:98856:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.371277:0:98856:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.371284:0:98856:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695650.371302:0:98856:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.371337:0:98856:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.371376:0:98856:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.371384:0:98856:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.371394:0:98856:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695650.371396:0:98856:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328d70 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x51d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edd36 expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.371414:0:98856:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d69 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edd1a expref: -99 pid: 98856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695650.371428:0:98856:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.371436:0:98856:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce3480 x1874531134025344/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.371458:0:98856:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98856:1874531134025344:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695650.371552:0:98856:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.380546:0:98856:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98856:1874531134025344:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695650.380589:0:98856:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x51d:0x0](ffff9d85b3cdf448) 00000080:00200000:2.0:1787695650.477890:0:98860:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.477922:0:98860:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.477939:0:98860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695650.478007:0:98860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d93 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.478018:0:98860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d93 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.478028:0:98860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328d93 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.478055:0:98860:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98860:1874531134026752:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.478142:0:98860:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.547592:0:98862:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2df80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98862:1874531134027648:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.547695:0:98862:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.547711:0:98862:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.547726:0:98862:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.547740:0:98862:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.547756:0:98862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.547766:0:98862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.547773:0:98862:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.547787:0:98862:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2df80 x1874531134027648/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.547803:0:98862:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.547826:0:98862:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.547833:0:98862:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.548141:0:98862:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.548150:0:98862:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328da8 00000080:00010000:2.0:1787695650.548155:0:98862:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.548159:0:98862:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328da8 from it ffffb5d40411bc20 00010000:00010000:2.0:1787695650.548162:0:98862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.548172:0:98862:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.548200:0:98862:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.548224:0:98862:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.548228:0:98862:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.548249:0:98862:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.548294:0:98862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98862 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.548301:0:98862:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98862 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.548310:0:98862:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98862 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.548332:0:98862:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98862:1874531134027776:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.548457:0:98862:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.552456:0:98862:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98862:1874531134027776:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.552498:0:98862:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552525:0:98862:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 4/1,0 mode: --/PR res: [0x200000401:0x520:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552533:0:98862:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 4/1,0 mode: --/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552543:0:98862:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552554:0:98862:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552565:0:98862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552572:0:98862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552580:0:98862:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.552591:0:98862:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce0000 x1874531134027776/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.552603:0:98862:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552614:0:98862:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1312 (0 0) 00000080:00200000:2.0:1787695650.552620:0:98862:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695650.552671:0:98862:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x520:0x0](ffff9d85b3cdf448) 00000080:00200000:2.0:1787695650.552681:0:98862:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x520:0x0], updating i_size 512 00000080:00010000:2.0:1787695650.552689:0:98862:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x520:0x0](ffff9d85b3cdf448) for lock 0x678ae32e95328daf 00000080:00010000:2.0:1787695650.552701:0:98862:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x520:0x0](ffff9d85b3cdf448) 00000080:00010000:2.0:1787695650.552704:0:98862:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328daf from it ffffb5d40411be08 00010000:00010000:2.0:1787695650.552707:0:98862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.552716:0:98862:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.552751:0:98862:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.552769:0:98862:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.552778:0:98862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.552789:0:98862:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.552795:0:98862:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.552799:0:98862:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.552802:0:98862:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328da8 00000080:00010000:2.0:1787695650.552806:0:98862:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328da8 from it ffffb5d40411bd90 00010000:00010000:2.0:1787695650.552808:0:98862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.552815:0:98862:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.552822:0:98862:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.552829:0:98862:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695650.552880:0:98862:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.553127:0:98862:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.553169:0:98862:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 1/0,0 mode: --/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.553176:0:98862:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.553181:0:98862:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695650.553184:0:98862:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328daf lrc: 0/0,0 mode: --/PR res: [0x200000401:0x520:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ede47 expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.553201:0:98862:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328da8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ede2b expref: -99 pid: 98862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695650.553216:0:98862:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.553223:0:98862:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce0000 x1874531134028032/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.553252:0:98862:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98862:1874531134028032:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695650.553333:0:98862:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.559279:0:98862:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98862:1874531134028032:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695650.559332:0:98862:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x520:0x0](ffff9d85b3cdf448) 00000080:00200000:2.0:1787695650.743439:0:98869:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.743477:0:98869:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.743540:0:98869:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98869 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.743551:0:98869:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98869 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.743563:0:98869:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98869 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.743595:0:98869:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98869:1874531134030848:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695650.743711:0:98869:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.747498:0:98869:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98869:1874531134030848:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695650.747525:0:98869:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747538:0:98869:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747549:0:98869:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747564:0:98869:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747578:0:98869:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747585:0:98869:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747591:0:98869:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.747602:0:98869:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce0380 x1874531134030848/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.747614:0:98869:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747694:0:98869:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.747702:0:98869:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.747729:0:98869:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.747735:0:98869:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328df5 00000080:00010000:2.0:1787695650.747740:0:98869:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.747743:0:98869:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328df5 from it ffffb5d404133be0 00010000:00010000:2.0:1787695650.747746:0:98869:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.747758:0:98869:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.747782:0:98869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.747801:0:98869:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695650.747806:0:98869:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.747819:0:98869:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.747828:0:98869:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.747839:0:98869:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.747845:0:98869:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.748129:0:98869:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.748133:0:98869:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328df5 00000080:00010000:2.0:1787695650.748138:0:98869:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328df5 from it ffffb5d404133ce0 00010000:00010000:2.0:1787695650.748142:0:98869:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.748149:0:98869:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.748159:0:98869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.748165:0:98869:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.748172:0:98869:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.748180:0:98869:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.748188:0:98869:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.748194:0:98869:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.748197:0:98869:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.748199:0:98869:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328df5 00000080:00010000:2.0:1787695650.748202:0:98869:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328df5 from it ffffb5d404133dc8 00010000:00010000:2.0:1787695650.748205:0:98869:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.748212:0:98869:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695650.748219:0:98869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.748223:0:98869:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695650.748252:0:98869:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.748295:0:98869:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.748301:0:98869:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695650.748304:0:98869:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328df5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832edf7b expref: -99 pid: 98869 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695650.748327:0:98869:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.748335:0:98869:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce0380 x1874531134031104/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.748359:0:98869:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98869:1874531134031104:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695650.748497:0:98869:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.753642:0:98869:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98869:1874531134031104:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695650.753690:0:98869:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x524:0x0](ffff9d85b3cdf448) 00000080:00200000:2.0:1787695650.753697:0:98869:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x524:0x0], updating i_size 512 00000080:00200000:2.0:1787695650.760247:0:98870:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.760276:0:98870:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.760290:0:98870:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695650.760342:0:98870:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98870 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.760351:0:98870:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98870 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.760362:0:98870:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98870 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.760387:0:98870:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98870:1874531134031232:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.760483:0:98870:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.763784:0:98870:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98870:1874531134031232:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.763812:0:98870:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.763823:0:98870:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.763840:0:98870:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.763905:0:98870:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.764201:0:98870:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.764209:0:98870:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.764216:0:98870:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.764228:0:98870:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1180 x1874531134031232/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.764243:0:98870:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328dfc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832edf97 expref: -99 pid: 98870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.764257:0:98870:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.764264:0:98870:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.798450:0:98871:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x525:0x0](ffff9d85b3cdba88) 00000080:00200000:2.0:1787695650.798470:0:98871:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x525:0x0], updating i_size 512 00000080:00200000:2.0:1787695650.903163:0:98876:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.903200:0:98876:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.903216:0:98876:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695650.903275:0:98876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98876 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.903285:0:98876:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98876 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.903295:0:98876:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98876 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.903321:0:98876:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98876:1874531134033920:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.903466:0:98876:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.906560:0:98876:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98876:1874531134033920:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.906584:0:98876:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.906596:0:98876:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.906607:0:98876:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.906618:0:98876:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.906718:0:98876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.906729:0:98876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.906737:0:98876:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.906749:0:98876:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce0a80 x1874531134033920/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.906763:0:98876:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.906776:0:98876:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.906783:0:98876:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.906810:0:98876:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.906816:0:98876:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328e3b 00000080:00010000:2.0:1787695650.908816:0:98876:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.908821:0:98876:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e3b from it ffffb5d40412bc20 00010000:00010000:2.0:1787695650.908827:0:98876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.908841:0:98876:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.908892:0:98876:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.909115:0:98876:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.909121:0:98876:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.909142:0:98876:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.909190:0:98876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98876 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.909200:0:98876:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98876 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.909208:0:98876:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98876 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.909232:0:98876:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98876:1874531134034048:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.909310:0:98876:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.912510:0:98876:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98876:1874531134034048:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.912536:0:98876:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912557:0:98876:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x527:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912566:0:98876:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912576:0:98876:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912587:0:98876:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912599:0:98876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912611:0:98876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912618:0:98876:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.912771:0:98876:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce0a80 x1874531134034048/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.912785:0:98876:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.912795:0:98876:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1319 (0 0) 00000080:00200000:2.0:1787695650.912802:0:98876:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695650.912831:0:98876:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x527:0x0](ffff9d85b3cd9988) 00000080:00200000:2.0:1787695650.912840:0:98876:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x527:0x0], updating i_size 512 00000080:00010000:2.0:1787695650.912847:0:98876:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x527:0x0](ffff9d85b3cd9988) for lock 0x678ae32e95328e42 00000080:00010000:2.0:1787695650.913103:0:98876:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x527:0x0](ffff9d85b3cd9988) 00000080:00010000:2.0:1787695650.913108:0:98876:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e42 from it ffffb5d40412be08 00010000:00010000:2.0:1787695650.913112:0:98876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.913122:0:98876:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.913151:0:98876:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.913166:0:98876:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.913176:0:98876:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.913186:0:98876:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.913192:0:98876:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.913196:0:98876:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.913199:0:98876:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328e3b 00000080:00010000:2.0:1787695650.913202:0:98876:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e3b from it ffffb5d40412bd90 00010000:00010000:2.0:1787695650.913205:0:98876:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.913211:0:98876:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.913220:0:98876:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.913228:0:98876:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695650.913246:0:98876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.913279:0:98876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.913311:0:98876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.913319:0:98876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.913330:0:98876:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695650.913333:0:98876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e42 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x527:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee0c4 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.913349:0:98876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e3b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee0a8 expref: -99 pid: 98876 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695650.913364:0:98876:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.913370:0:98876:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce0a80 x1874531134034304/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.913392:0:98876:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98876:1874531134034304:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695650.913512:0:98876:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.918484:0:98876:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98876:1874531134034304:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695650.918523:0:98876:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x527:0x0](ffff9d85b3cd9988) 00000100:00100000:2.0:1787695650.978503:0:98879:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3bb80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98879:1874531134035328:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695650.978534:0:98879:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978547:0:98879:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978558:0:98879:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978569:0:98879:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978582:0:98879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978589:0:98879:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978596:0:98879:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.978608:0:98879:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3bb80 x1874531134035328/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.978621:0:98879:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978751:0:98879:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.978758:0:98879:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.978790:0:98879:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.978797:0:98879:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328e5e 00000080:00010000:2.0:1787695650.978803:0:98879:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.978806:0:98879:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e5e from it ffffb5d40412bbe0 00010000:00010000:2.0:1787695650.978809:0:98879:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.978817:0:98879:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.978844:0:98879:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.979344:0:98879:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695650.979350:0:98879:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.979368:0:98879:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.979378:0:98879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.979389:0:98879:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.979395:0:98879:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.979400:0:98879:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.979402:0:98879:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328e5e 00000080:00010000:2.0:1787695650.979406:0:98879:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e5e from it ffffb5d40412bce0 00010000:00010000:2.0:1787695650.979409:0:98879:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695650.979415:0:98879:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.979423:0:98879:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.979429:0:98879:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695650.979435:0:98879:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.979441:0:98879:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.979464:0:98879:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695650.979471:0:98879:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695650.979475:0:98879:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695650.979478:0:98879:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328e5e 00000080:00010000:2.0:1787695650.979481:0:98879:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e5e from it ffffb5d40412bdc8 00010000:00010000:2.0:1787695650.979484:0:98879:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695650.979492:0:98879:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695650.979499:0:98879:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695650.979505:0:98879:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695650.979534:0:98879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.979575:0:98879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695650.979583:0:98879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695650.979586:0:98879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328e5e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee142 expref: -99 pid: 98879 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695650.979608:0:98879:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695650.979616:0:98879:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce1880 x1874531134035584/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695650.979738:0:98879:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98879:1874531134035584:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695650.979893:0:98879:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.986193:0:98879:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98879:1874531134035584:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695650.986242:0:98879:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x529:0x0](ffff9d85b3cd9988) 00000080:00200000:2.0:1787695650.986249:0:98879:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x529:0x0], updating i_size 512 00000080:00200000:2.0:1787695650.994812:0:98880:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695650.994835:0:98880:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695650.994848:0:98880:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695650.995177:0:98880:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98880 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.995188:0:98880:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98880 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.995198:0:98880:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98880 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.995223:0:98880:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98880:1874531134035712:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.995352:0:98880:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695650.999120:0:98880:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98880:1874531134035712:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695650.999145:0:98880:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999158:0:98880:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999173:0:98880:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999183:0:98880:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999195:0:98880:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999203:0:98880:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999210:0:98880:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695650.999221:0:98880:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce3800 x1874531134035712/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695650.999235:0:98880:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999250:0:98880:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695650.999257:0:98880:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695650.999283:0:98880:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695650.999289:0:98880:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328e65 00000080:00010000:2.0:1787695650.999294:0:98880:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695650.999297:0:98880:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e65 from it ffffb5d4040f3c20 00010000:00010000:2.0:1787695650.999301:0:98880:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999309:0:98880:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695650.999329:0:98880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695650.999351:0:98880:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695650.999355:0:98880:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695650.999373:0:98880:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695650.999409:0:98880:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98880 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999416:0:98880:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98880 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695650.999424:0:98880:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98880 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695650.999442:0:98880:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98880:1874531134035840:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695650.999563:0:98880:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.002577:0:98880:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98880:1874531134035840:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.002603:0:98880:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002703:0:98880:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 4/1,0 mode: --/PR res: [0x200000401:0x529:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002713:0:98880:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 4/1,0 mode: --/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002725:0:98880:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002736:0:98880:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002748:0:98880:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002755:0:98880:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002761:0:98880:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.002772:0:98880:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce3800 x1874531134035840/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695667 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.002784:0:98880:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.002793:0:98880:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1321 (0 0) 00000080:00200000:2.0:1787695651.002800:0:98880:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695651.002830:0:98880:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x529:0x0](ffff9d85b3cdcb08) 00000080:00200000:2.0:1787695651.002838:0:98880:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x529:0x0], updating i_size 512 00000080:00010000:2.0:1787695651.002844:0:98880:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x529:0x0](ffff9d85b3cdcb08) for lock 0x678ae32e95328e6c 00000080:00010000:2.0:1787695651.003379:0:98880:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x529:0x0](ffff9d85b3cdcb08) 00000080:00010000:2.0:1787695651.003385:0:98880:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e6c from it ffffb5d4040f3e08 00010000:00010000:2.0:1787695651.003390:0:98880:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.003399:0:98880:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.003428:0:98880:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.003443:0:98880:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.003465:0:98880:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.003474:0:98880:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.003481:0:98880:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.003486:0:98880:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.003489:0:98880:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328e65 00000080:00010000:2.0:1787695651.003492:0:98880:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e65 from it ffffb5d4040f3d90 00010000:00010000:2.0:1787695651.003495:0:98880:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.003501:0:98880:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.003510:0:98880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.003517:0:98880:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695651.003537:0:98880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.003570:0:98880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.003606:0:98880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 1/0,0 mode: --/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.003613:0:98880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.003619:0:98880:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695651.003622:0:98880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e6c lrc: 0/0,0 mode: --/PR res: [0x200000401:0x529:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee17a expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.003649:0:98880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328e65 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee15e expref: -99 pid: 98880 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695651.003666:0:98880:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.003673:0:98880:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce3800 x1874531134036096/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.003696:0:98880:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98880:1874531134036096:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695651.003771:0:98880:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.008188:0:98880:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98880:1874531134036096:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.008234:0:98880:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x529:0x0](ffff9d85b3cdcb08) 00000080:00200000:2.0:1787695651.017710:0:98881:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.017737:0:98881:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.017801:0:98881:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98881 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.017813:0:98881:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98881 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.017824:0:98881:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98881 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.018359:0:98881:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98881:1874531134036224:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.018512:0:98881:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.022394:0:98881:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98881:1874531134036224:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.022423:0:98881:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022435:0:98881:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022446:0:98881:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022466:0:98881:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022480:0:98881:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022488:0:98881:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022499:0:98881:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.022512:0:98881:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce2d80 x1874531134036224/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.022526:0:98881:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022539:0:98881:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.022545:0:98881:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.022573:0:98881:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.022579:0:98881:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328e73 00000080:00010000:2.0:1787695651.022583:0:98881:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.022586:0:98881:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e73 from it ffffb5d40413bbe0 00010000:00010000:2.0:1787695651.022589:0:98881:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.022597:0:98881:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.022620:0:98881:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.022654:0:98881:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695651.022659:0:98881:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.022673:0:98881:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.022681:0:98881:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.022697:0:98881:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.022703:0:98881:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.022708:0:98881:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.022710:0:98881:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328e73 00000080:00010000:2.0:1787695651.022714:0:98881:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e73 from it ffffb5d40413bce0 00010000:00010000:2.0:1787695651.022717:0:98881:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.022724:0:98881:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.022733:0:98881:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.022739:0:98881:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.022745:0:98881:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.022751:0:98881:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.022759:0:98881:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.022765:0:98881:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.022768:0:98881:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.022770:0:98881:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328e73 00000080:00010000:2.0:1787695651.022773:0:98881:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328e73 from it ffffb5d40413bdc8 00010000:00010000:2.0:1787695651.022777:0:98881:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.022783:0:98881:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695651.022791:0:98881:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.022796:0:98881:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695651.022828:0:98881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.023160:0:98881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.023168:0:98881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695651.023171:0:98881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328e73 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee19d expref: -99 pid: 98881 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695651.023196:0:98881:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.023204:0:98881:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce2d80 x1874531134036480/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.023225:0:98881:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98881:1874531134036480:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695651.023301:0:98881:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000080:00200000:2.0:1787695651.038217:0:98882:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.038246:0:98882:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.038260:0:98882:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695651.038314:0:98882:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e7a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98882 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.038329:0:98882:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e7a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98882 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.038339:0:98882:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328e7a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98882 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.038365:0:98882:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98882:1874531134036608:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.038467:0:98882:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.071742:0:98883:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85ae913100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98883:1874531134037120:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.071773:0:98883:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.071786:0:98883:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.071797:0:98883:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.071809:0:98883:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.071820:0:98883:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.071828:0:98883:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.071841:0:98883:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.075156:0:98883:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85ae913100 x1874531134037120/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.075185:0:98883:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936fc00/0x678ae32e95328e88 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee1f8 expref: -99 pid: 98883 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.075202:0:98883:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.075209:0:98883:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000100:00100000:2.0:1787695651.152812:0:98886:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98886:1874531134038528:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.152836:0:98886:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.152885:0:98886:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 4/1,0 mode: --/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.152894:0:98886:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 4/1,0 mode: --/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.152915:0:98886:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.152927:0:98886:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.152945:0:98886:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.152952:0:98886:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.152959:0:98886:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.152971:0:98886:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2f480 x1874531134038528/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.153160:0:98886:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.153171:0:98886:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1324 (0 0) 00000080:00200000:2.0:1787695651.153177:0:98886:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695651.153206:0:98886:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x52c:0x0](ffff9d85b3cdcb08) 00000080:00200000:2.0:1787695651.153214:0:98886:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x52c:0x0], updating i_size 512 00000080:00010000:2.0:1787695651.153221:0:98886:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x52c:0x0](ffff9d85b3cdcb08) for lock 0x678ae32e95328eab 00000080:00010000:2.0:1787695651.153232:0:98886:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x52c:0x0](ffff9d85b3cdcb08) 00000080:00010000:2.0:1787695651.153235:0:98886:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328eab from it ffffb5d4040f3e08 00010000:00010000:2.0:1787695651.153238:0:98886:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.153247:0:98886:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.153282:0:98886:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.153299:0:98886:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.153309:0:98886:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95328ea4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee26f expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.153319:0:98886:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95328ea4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee26f expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.153326:0:98886:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.153331:0:98886:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.153334:0:98886:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ea4 00000080:00010000:2.0:1787695651.153337:0:98886:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ea4 from it ffffb5d4040f3d90 00010000:00010000:2.0:1787695651.153340:0:98886:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95328ea4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee26f expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.153347:0:98886:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95328ea4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee26f expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.153354:0:98886:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.153362:0:98886:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695651.153382:0:98886:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95328ea4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee26f expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.153416:0:98886:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.153460:0:98886:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 1/0,0 mode: --/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.153469:0:98886:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95328ea4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee26f expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.153475:0:98886:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695651.153478:0:98886:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031000/0x678ae32e95328eab lrc: 0/0,0 mode: --/PR res: [0x200000401:0x52c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee28b expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.153496:0:98886:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95328ea4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee26f expref: -99 pid: 98886 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695651.153515:0:98886:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.153523:0:98886:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce2680 x1874531134038784/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.153547:0:98886:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98886:1874531134038784:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695651.153753:0:98886:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.160365:0:98886:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98886:1874531134038784:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.160412:0:98886:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x52c:0x0](ffff9d85b3cdcb08) 00000080:00200000:2.0:1787695651.171264:0:98887:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.171291:0:98887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.171358:0:98887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98887 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.171371:0:98887:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98887 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.171380:0:98887:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98887 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.171406:0:98887:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98887:1874531134038912:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.171515:0:98887:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.175359:0:98887:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98887:1874531134038912:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.175388:0:98887:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175400:0:98887:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175411:0:98887:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175421:0:98887:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175432:0:98887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175439:0:98887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175445:0:98887:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.175468:0:98887:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1500 x1874531134038912/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.175485:0:98887:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175497:0:98887:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.175503:0:98887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.175532:0:98887:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.175539:0:98887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328eb2 00000080:00010000:2.0:1787695651.175544:0:98887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.175547:0:98887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328eb2 from it ffffb5d404133be0 00010000:00010000:2.0:1787695651.175550:0:98887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.175558:0:98887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.175582:0:98887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.175605:0:98887:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695651.175609:0:98887:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.175621:0:98887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.175641:0:98887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.175651:0:98887:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.175658:0:98887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.175661:0:98887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.175664:0:98887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328eb2 00000080:00010000:2.0:1787695651.175668:0:98887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328eb2 from it ffffb5d404133ce0 00010000:00010000:2.0:1787695651.175671:0:98887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.175677:0:98887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.175685:0:98887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.175692:0:98887:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.175698:0:98887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.175704:0:98887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.175712:0:98887:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.175718:0:98887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.175722:0:98887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.175724:0:98887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328eb2 00000080:00010000:2.0:1787695651.175727:0:98887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328eb2 from it ffffb5d404133dc8 00010000:00010000:2.0:1787695651.175731:0:98887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.175738:0:98887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695651.175747:0:98887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.175753:0:98887:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695651.175785:0:98887:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.175825:0:98887:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.175833:0:98887:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695651.175835:0:98887:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328eb2 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee2ae expref: -99 pid: 98887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695651.176256:0:98887:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.176266:0:98887:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce1500 x1874531134039168/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.176289:0:98887:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98887:1874531134039168:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695651.229319:0:98889:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.229344:0:98889:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.229404:0:98889:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98889 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.229416:0:98889:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98889 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.229426:0:98889:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98889 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.229462:0:98889:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98889:1874531134039808:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.229605:0:98889:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.233547:0:98889:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98889:1874531134039808:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.233578:0:98889:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233591:0:98889:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233601:0:98889:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233611:0:98889:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233718:0:98889:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233729:0:98889:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233736:0:98889:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.233748:0:98889:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce3b80 x1874531134039808/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.233762:0:98889:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233775:0:98889:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.233782:0:98889:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.233812:0:98889:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.233818:0:98889:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ec7 00000080:00010000:2.0:1787695651.233824:0:98889:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.233827:0:98889:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ec7 from it ffffb5d40414bbe0 00010000:00010000:2.0:1787695651.233830:0:98889:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.233839:0:98889:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.233877:0:98889:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.234058:0:98889:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695651.234065:0:98889:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.234081:0:98889:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.234091:0:98889:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.234102:0:98889:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.234109:0:98889:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.234113:0:98889:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.234116:0:98889:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ec7 00000080:00010000:2.0:1787695651.234120:0:98889:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ec7 from it ffffb5d40414bce0 00010000:00010000:2.0:1787695651.234123:0:98889:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.234130:0:98889:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.234140:0:98889:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.234147:0:98889:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.234154:0:98889:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.234159:0:98889:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.234167:0:98889:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.234172:0:98889:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.234176:0:98889:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.234178:0:98889:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ec7 00000080:00010000:2.0:1787695651.234181:0:98889:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ec7 from it ffffb5d40414bdc8 00010000:00010000:2.0:1787695651.234184:0:98889:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.234191:0:98889:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695651.234198:0:98889:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.234204:0:98889:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695651.234234:0:98889:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.234279:0:98889:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.234289:0:98889:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695651.234292:0:98889:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328ec7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee309 expref: -99 pid: 98889 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695651.234314:0:98889:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.234321:0:98889:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce3b80 x1874531134040064/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.234345:0:98889:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98889:1874531134040064:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695651.234644:0:98889:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.241220:0:98889:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98889:1874531134040064:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695651.241272:0:98889:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x52e:0x0](ffff9d85b3cdcb08) 00000080:00200000:2.0:1787695651.241280:0:98889:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x52e:0x0], updating i_size 512 00010000:00010000:2.0:1787695651.261652:0:98890:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261662:0:98890:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261673:0:98890:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261684:0:98890:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261711:0:98890:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261719:0:98890:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261726:0:98890:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.261737:0:98890:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fd880 x1874531134040320/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.261749:0:98890:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261758:0:98890:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1326 (0 0) 00000080:00200000:2.0:1787695651.261764:0:98890:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695651.261797:0:98890:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x52e:0x0](ffff9d85b3cdd348) 00000080:00200000:2.0:1787695651.261805:0:98890:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x52e:0x0], updating i_size 512 00000080:00010000:2.0:1787695651.261811:0:98890:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x52e:0x0](ffff9d85b3cdd348) for lock 0x678ae32e95328ed5 00000080:00010000:2.0:1787695651.261822:0:98890:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x52e:0x0](ffff9d85b3cdd348) 00000080:00010000:2.0:1787695651.261825:0:98890:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ed5 from it ffffb5d404153e08 00010000:00010000:2.0:1787695651.261828:0:98890:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.261837:0:98890:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.262124:0:98890:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.262144:0:98890:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.262155:0:98890:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ea00/0x678ae32e95328ece lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee325 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.262166:0:98890:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ea00/0x678ae32e95328ece lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee325 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.262173:0:98890:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.262178:0:98890:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.262180:0:98890:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ece 00000080:00010000:2.0:1787695651.262184:0:98890:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ece from it ffffb5d404153d90 00010000:00010000:2.0:1787695651.262187:0:98890:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ea00/0x678ae32e95328ece lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee325 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.262194:0:98890:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ea00/0x678ae32e95328ece lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee325 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.262202:0:98890:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.262210:0:98890:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695651.262232:0:98890:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ea00/0x678ae32e95328ece lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee325 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.262270:0:98890:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.262305:0:98890:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.262313:0:98890:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ea00/0x678ae32e95328ece lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee325 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.262324:0:98890:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695651.262327:0:98890:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e95328ed5 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x52e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee341 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.262341:0:98890:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ea00/0x678ae32e95328ece lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee325 expref: -99 pid: 98890 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695651.262357:0:98890:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.262365:0:98890:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce2a00 x1874531134040576/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.262386:0:98890:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98890:1874531134040576:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695651.262505:0:98890:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.267618:0:98890:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98890:1874531134040576:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.267674:0:98890:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x52e:0x0](ffff9d85b3cdd348) 00000080:00200000:2.0:1787695651.277752:0:98891:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.277778:0:98891:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.278337:0:98891:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98891 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.278349:0:98891:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98891 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.278359:0:98891:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98891 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.278385:0:98891:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98891:1874531134040704:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.278529:0:98891:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.281529:0:98891:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98891:1874531134040704:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.281559:0:98891:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281572:0:98891:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281584:0:98891:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281594:0:98891:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281606:0:98891:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281614:0:98891:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281620:0:98891:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.281631:0:98891:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce0700 x1874531134040704/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.281644:0:98891:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281661:0:98891:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.281668:0:98891:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.281694:0:98891:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.281701:0:98891:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328edc 00000080:00010000:2.0:1787695651.281705:0:98891:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.281708:0:98891:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328edc from it ffffb5d40411bbe0 00010000:00010000:2.0:1787695651.281712:0:98891:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.281720:0:98891:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.281744:0:98891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.281770:0:98891:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695651.281774:0:98891:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.281786:0:98891:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.281794:0:98891:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.281804:0:98891:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.281811:0:98891:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.281815:0:98891:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.281818:0:98891:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328edc 00000080:00010000:2.0:1787695651.282372:0:98891:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328edc from it ffffb5d40411bce0 00010000:00010000:2.0:1787695651.282378:0:98891:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.282387:0:98891:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.282398:0:98891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.282407:0:98891:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.282766:0:98891:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.282777:0:98891:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.282786:0:98891:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.282792:0:98891:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.282796:0:98891:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.282799:0:98891:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328edc 00000080:00010000:2.0:1787695651.282802:0:98891:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328edc from it ffffb5d40411bdc8 00010000:00010000:2.0:1787695651.282806:0:98891:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.282812:0:98891:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695651.283111:0:98891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.283119:0:98891:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695651.283152:0:98891:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.283402:0:98891:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.283411:0:98891:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695651.283414:0:98891:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328edc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee364 expref: -99 pid: 98891 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695651.283437:0:98891:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.283458:0:98891:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce0700 x1874531134040960/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.283482:0:98891:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98891:1874531134040960:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695651.283621:0:98891:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.289234:0:98891:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98891:1874531134040960:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695651.289286:0:98891:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x52f:0x0](ffff9d85b3cdd348) 00000080:00200000:2.0:1787695651.289294:0:98891:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x52f:0x0], updating i_size 512 00010000:00010000:0.0:1787695651.300597:0:98892:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.300610:0:98892:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.300636:0:98892:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695651.300659:0:98892:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695651.300664:0:98892:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695651.300682:0:98892:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695651.300722:0:98892:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98892 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.300729:0:98892:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98892 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.300737:0:98892:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98892 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695651.300755:0:98892:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98892:1874531134041216:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695651.300888:0:98892:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1e40 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.304363:0:98892:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98892:1874531134041216:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695651.304390:0:98892:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304413:0:98892:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 4/1,0 mode: --/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304422:0:98892:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 4/1,0 mode: --/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304433:0:98892:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304444:0:98892:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304469:0:98892:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304476:0:98892:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304483:0:98892:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695651.304494:0:98892:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39880 x1874531134041216/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695651.304506:0:98892:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304516:0:98892:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1327 (0 0) 00000080:00200000:0.0:1787695651.304524:0:98892:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695651.304552:0:98892:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x52f:0x0](ffff9d85b3cd5b88) 00000080:00200000:0.0:1787695651.304560:0:98892:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x52f:0x0], updating i_size 512 00000080:00010000:0.0:1787695651.304566:0:98892:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x52f:0x0](ffff9d85b3cd5b88) for lock 0x678ae32e95328eea 00000080:00010000:0.0:1787695651.304576:0:98892:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x52f:0x0](ffff9d85b3cd5b88) 00000080:00010000:0.0:1787695651.304579:0:98892:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328eea from it ffffb5d4040fbe08 00010000:00010000:0.0:1787695651.304582:0:98892:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.304590:0:98892:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.304617:0:98892:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695651.304715:0:98892:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.304727:0:98892:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.304738:0:98892:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.304744:0:98892:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695651.304749:0:98892:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695651.304752:0:98892:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ee3 00000080:00010000:0.0:1787695651.304755:0:98892:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ee3 from it ffffb5d4040fbd90 00010000:00010000:0.0:1787695651.304758:0:98892:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.304764:0:98892:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695651.304771:0:98892:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695651.304779:0:98892:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695651.304801:0:98892:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.304833:0:98892:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.305182:0:98892:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 1/0,0 mode: --/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.305192:0:98892:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.305197:0:98892:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695651.305200:0:98892:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e95328eea lrc: 0/0,0 mode: --/PR res: [0x200000401:0x52f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee39c expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.305217:0:98892:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e95328ee3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee380 expref: -99 pid: 98892 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695651.305234:0:98892:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695651.305242:0:98892:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f39880 x1874531134041472/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695651.305264:0:98892:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98892:1874531134041472:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695651.305376:0:98892:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1e40 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.309669:0:98892:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98892:1874531134041472:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695651.309712:0:98892:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x52f:0x0](ffff9d85b3cd5b88) 00000080:00200000:0.0:1787695651.320260:0:98893:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695651.320285:0:98893:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.320350:0:98893:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98893 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.320362:0:98893:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98893 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.320373:0:98893:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98893 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695651.320397:0:98893:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98893:1874531134041600:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695651.320546:0:98893:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1e40 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.323892:0:98893:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98893:1874531134041600:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695651.324083:0:98893:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324096:0:98893:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324107:0:98893:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324117:0:98893:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324128:0:98893:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324135:0:98893:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324146:0:98893:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695651.324158:0:98893:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f38e00 x1874531134041600/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695651.324172:0:98893:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324184:0:98893:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695651.324191:0:98893:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695651.324217:0:98893:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:0.0:1787695651.324223:0:98893:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ef1 00000080:00010000:0.0:1787695651.324228:0:98893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695651.324231:0:98893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ef1 from it ffffb5d40415bbe0 00010000:00010000:0.0:1787695651.324234:0:98893:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.324244:0:98893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.324267:0:98893:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695651.324287:0:98893:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695651.324292:0:98893:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695651.324304:0:98893:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.324313:0:98893:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.324328:0:98893:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.324334:0:98893:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695651.324338:0:98893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695651.324341:0:98893:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ef1 00000080:00010000:0.0:1787695651.324345:0:98893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ef1 from it ffffb5d40415bce0 00010000:00010000:0.0:1787695651.324348:0:98893:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.324356:0:98893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695651.324365:0:98893:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695651.324371:0:98893:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695651.324377:0:98893:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.324382:0:98893:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.324391:0:98893:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695651.324397:0:98893:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695651.324401:0:98893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695651.324403:0:98893:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328ef1 00000080:00010000:0.0:1787695651.324407:0:98893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ef1 from it ffffb5d40415bdc8 00010000:00010000:0.0:1787695651.324411:0:98893:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.324417:0:98893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695651.324424:0:98893:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695651.324430:0:98893:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695651.324469:0:98893:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695651.324510:0:98893:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695651.324516:0:98893:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695651.324519:0:98893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95328ef1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee3bf expref: -99 pid: 98893 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695651.324543:0:98893:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695651.324550:0:98893:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f38e00 x1874531134041856/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695651.324570:0:98893:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98893:1874531134041856:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695651.324750:0:98893:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1e40 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.331843:0:98893:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98893:1874531134041856:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695651.331920:0:98893:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x530:0x0](ffff9d85b3cd5b88) 00000080:00200000:0.0:1787695651.331927:0:98893:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x530:0x0], updating i_size 512 00010000:00010000:0.0:1787695651.349567:0:98894:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 4/1,0 mode: --/PR res: [0x200000401:0x530:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349577:0:98894:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 4/1,0 mode: --/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349587:0:98894:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349598:0:98894:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349611:0:98894:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349619:0:98894:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349688:0:98894:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695651.349701:0:98894:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72ff480 x1874531134042112/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695651.349714:0:98894:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349723:0:98894:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1328 (0 0) 00000080:00200000:0.0:1787695651.349730:0:98894:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695651.349761:0:98894:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x530:0x0](ffff9d85b3cd6c08) 00000080:00200000:0.0:1787695651.349769:0:98894:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x530:0x0], updating i_size 512 00000080:00010000:0.0:1787695651.349776:0:98894:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x530:0x0](ffff9d85b3cd6c08) for lock 0x678ae32e95328eff 00000080:00010000:0.0:1787695651.349785:0:98894:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x530:0x0](ffff9d85b3cd6c08) 00000080:00010000:0.0:1787695651.349789:0:98894:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328eff from it ffffb5d404163e08 00010000:00010000:0.0:1787695651.349791:0:98894:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.349799:0:98894:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.349831:0:98894:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695651.349847:0:98894:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.350128:0:98894:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328ef8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3db expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.350138:0:98894:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328ef8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3db expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.350145:0:98894:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695651.350151:0:98894:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695651.350154:0:98894:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ef8 00000080:00010000:0.0:1787695651.350158:0:98894:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ef8 from it ffffb5d404163d90 00010000:00010000:0.0:1787695651.350162:0:98894:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328ef8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee3db expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.350169:0:98894:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328ef8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee3db expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695651.350178:0:98894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695651.350186:0:98894:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695651.350206:0:98894:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328ef8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee3db expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.350239:0:98894:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.350275:0:98894:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 1/0,0 mode: --/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.350282:0:98894:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328ef8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee3db expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.350289:0:98894:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695651.350292:0:98894:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f600/0x678ae32e95328eff lrc: 0/0,0 mode: --/PR res: [0x200000401:0x530:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee3f7 expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.350308:0:98894:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e95328ef8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee3db expref: -99 pid: 98894 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695651.350323:0:98894:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695651.350330:0:98894:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3b100 x1874531134042368/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695651.350352:0:98894:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98894:1874531134042368:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695651.350469:0:98894:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1e40 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.355345:0:98894:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98894:1874531134042368:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.355390:0:98894:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x530:0x0](ffff9d85b3cd6c08) 00000080:00200000:2.0:1787695651.362093:0:98895:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.362118:0:98895:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.362176:0:98895:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98895 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.362186:0:98895:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98895 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.362196:0:98895:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98895 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.362221:0:98895:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98895:1874531134042496:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.364386:0:98895:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000080:00200000:2.0:1787695651.385493:0:98896:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.385522:0:98896:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.385536:0:98896:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695651.385589:0:98896:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98896 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.385599:0:98896:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98896 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.385608:0:98896:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98896 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.385692:0:98896:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98896:1874531134042880:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.385804:0:98896:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.388510:0:98896:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98896:1874531134042880:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.388538:0:98896:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388550:0:98896:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388562:0:98896:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388572:0:98896:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388585:0:98896:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388592:0:98896:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388600:0:98896:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.388612:0:98896:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1c00 x1874531134042880/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.388639:0:98896:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388654:0:98896:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.388661:0:98896:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.388689:0:98896:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.388695:0:98896:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328f0d 00000080:00010000:2.0:1787695651.388700:0:98896:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.388703:0:98896:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f0d from it ffffb5d40416bc20 00010000:00010000:2.0:1787695651.388706:0:98896:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388714:0:98896:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.388740:0:98896:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.388762:0:98896:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695651.388767:0:98896:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695651.388785:0:98896:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695651.388825:0:98896:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98896 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388833:0:98896:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98896 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.388841:0:98896:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98896 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.389131:0:98896:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98896:1874531134043008:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.389218:0:98896:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.392231:0:98896:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98896:1874531134043008:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.392256:0:98896:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392277:0:98896:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x531:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392286:0:98896:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392296:0:98896:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392308:0:98896:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392319:0:98896:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392327:0:98896:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392335:0:98896:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.392346:0:98896:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1c00 x1874531134043008/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.392359:0:98896:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392368:0:98896:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1329 (0 0) 00000080:00200000:2.0:1787695651.392373:0:98896:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695651.392436:0:98896:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x531:0x0](ffff9d85b3cf21c8) 00000080:00200000:2.0:1787695651.392455:0:98896:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x531:0x0], updating i_size 512 00000080:00010000:2.0:1787695651.392464:0:98896:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x531:0x0](ffff9d85b3cf21c8) for lock 0x678ae32e95328f14 00000080:00010000:2.0:1787695651.392475:0:98896:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x531:0x0](ffff9d85b3cf21c8) 00000080:00010000:2.0:1787695651.392478:0:98896:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f14 from it ffffb5d40416be08 00010000:00010000:2.0:1787695651.392482:0:98896:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.392496:0:98896:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.392529:0:98896:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.392546:0:98896:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.392556:0:98896:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.392566:0:98896:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.392572:0:98896:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.392577:0:98896:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.392580:0:98896:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328f0d 00000080:00010000:2.0:1787695651.392583:0:98896:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f0d from it ffffb5d40416bd90 00010000:00010000:2.0:1787695651.392586:0:98896:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.392592:0:98896:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.392600:0:98896:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.392608:0:98896:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695651.392711:0:98896:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.392746:0:98896:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.392784:0:98896:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.392792:0:98896:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.392797:0:98896:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695651.392800:0:98896:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f14 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x531:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee452 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.392817:0:98896:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95328f0d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee436 expref: -99 pid: 98896 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695651.392832:0:98896:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.392839:0:98896:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce1c00 x1874531134043264/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.393088:0:98896:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98896:1874531134043264:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695651.393165:0:98896:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.398317:0:98896:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98896:1874531134043264:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.398367:0:98896:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x531:0x0](ffff9d85b3cf21c8) 00000080:00200000:2.0:1787695651.408507:0:98897:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.408536:0:98897:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.408598:0:98897:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98897 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.408609:0:98897:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98897 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.408620:0:98897:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98897 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.409354:0:98897:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98897:1874531134043392:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.409442:0:98897:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.412737:0:98897:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98897:1874531134043392:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.412764:0:98897:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412776:0:98897:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412786:0:98897:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412795:0:98897:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412807:0:98897:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412815:0:98897:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412821:0:98897:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.412838:0:98897:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce0e00 x1874531134043392/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.412887:0:98897:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412909:0:98897:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.412916:0:98897:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.412943:0:98897:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.412949:0:98897:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f1b 00000080:00010000:2.0:1787695651.412955:0:98897:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.412958:0:98897:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f1b from it ffffb5d404173be0 00010000:00010000:2.0:1787695651.412961:0:98897:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.412970:0:98897:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.413002:0:98897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.413024:0:98897:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695651.413029:0:98897:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.413041:0:98897:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.413049:0:98897:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.413060:0:98897:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.413072:0:98897:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.413076:0:98897:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.413079:0:98897:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f1b 00000080:00010000:2.0:1787695651.413082:0:98897:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f1b from it ffffb5d404173ce0 00010000:00010000:2.0:1787695651.413085:0:98897:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.413092:0:98897:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.413100:0:98897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.413106:0:98897:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.413112:0:98897:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.413118:0:98897:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.413125:0:98897:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.413131:0:98897:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.413134:0:98897:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.413136:0:98897:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f1b 00000080:00010000:2.0:1787695651.413139:0:98897:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f1b from it ffffb5d404173dc8 00010000:00010000:2.0:1787695651.413143:0:98897:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.413155:0:98897:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695651.413163:0:98897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.413168:0:98897:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695651.413197:0:98897:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.413238:0:98897:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.413245:0:98897:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695651.413248:0:98897:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95328f1b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee475 expref: -99 pid: 98897 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695651.413270:0:98897:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.413277:0:98897:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce0e00 x1874531134043648/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.413299:0:98897:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98897:1874531134043648:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695651.413364:0:98897:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.421649:0:98897:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98897:1874531134043648:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695651.421703:0:98897:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x532:0x0](ffff9d85b3cf21c8) 00000080:00200000:2.0:1787695651.421710:0:98897:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x532:0x0], updating i_size 512 00000100:00100000:2.0:1787695651.433522:0:98898:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9eae900 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.437275:0:98898:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fc700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98898:1874531134043776:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.437305:0:98898:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437317:0:98898:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437327:0:98898:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437338:0:98898:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437350:0:98898:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437357:0:98898:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437364:0:98898:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.437376:0:98898:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fc700 x1874531134043776/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.437389:0:98898:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437402:0:98898:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.437409:0:98898:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.437439:0:98898:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.437479:0:98898:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328f22 00000080:00010000:2.0:1787695651.437486:0:98898:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.437489:0:98898:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f22 from it ffffb5d40413bc20 00010000:00010000:2.0:1787695651.437492:0:98898:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437502:0:98898:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328f22 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee491 expref: -99 pid: 98898 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.437526:0:98898:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.437549:0:98898:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695651.437553:0:98898:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695651.437574:0:98898:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695651.437617:0:98898:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328f29 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98898 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.437842:0:98898:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328f29 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98898 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.438082:0:98898:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95328f29 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98898 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.438107:0:98898:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98898:1874531134043904:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.438230:0:98898:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000002:00010000:2.0:1787695651.464256:0:98899:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.464330:0:98899:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98899 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.464345:0:98899:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98899 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.464356:0:98899:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98899 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.464382:0:98899:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98899:1874531134044288:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.464485:0:98899:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.468107:0:98899:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98899:1874531134044288:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.468135:0:98899:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468148:0:98899:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468160:0:98899:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468170:0:98899:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468183:0:98899:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468192:0:98899:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468205:0:98899:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.468217:0:98899:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce3100 x1874531134044288/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.468231:0:98899:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468244:0:98899:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.468251:0:98899:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.468281:0:98899:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.468288:0:98899:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f30 00000080:00010000:2.0:1787695651.468294:0:98899:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.468297:0:98899:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f30 from it ffffb5d4040fbbe0 00010000:00010000:2.0:1787695651.468300:0:98899:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.468310:0:98899:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.468332:0:98899:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.468353:0:98899:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695651.468358:0:98899:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.468371:0:98899:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.468380:0:98899:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.468397:0:98899:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.468403:0:98899:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.468407:0:98899:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.468410:0:98899:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f30 00000080:00010000:2.0:1787695651.468414:0:98899:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f30 from it ffffb5d4040fbce0 00010000:00010000:2.0:1787695651.468417:0:98899:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.468423:0:98899:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.468431:0:98899:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.468438:0:98899:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.468456:0:98899:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.468462:0:98899:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.468471:0:98899:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.468477:0:98899:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.468481:0:98899:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.468484:0:98899:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f30 00000080:00010000:2.0:1787695651.468488:0:98899:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f30 from it ffffb5d4040fbdc8 00010000:00010000:2.0:1787695651.468491:0:98899:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.468498:0:98899:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695651.468506:0:98899:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.468512:0:98899:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695651.468541:0:98899:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.468584:0:98899:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.468592:0:98899:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695651.468595:0:98899:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f30 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee4d0 expref: -99 pid: 98899 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695651.468618:0:98899:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.468725:0:98899:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce3100 x1874531134044544/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.468750:0:98899:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98899:1874531134044544:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695651.469076:0:98899:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.474486:0:98899:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98899:1874531134044544:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695651.474539:0:98899:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x533:0x0](ffff9d85b3cf00c8) 00000080:00200000:2.0:1787695651.474547:0:98899:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x533:0x0], updating i_size 512 00000080:00200000:0.0:1787695651.481546:0:98900:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695651.481576:0:98900:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.481593:0:98900:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695651.481754:0:98900:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95328f37 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98900 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.481764:0:98900:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95328f37 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98900 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.481774:0:98900:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95328f37 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98900 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695651.481804:0:98900:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98900:1874531134044672:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695651.482203:0:98900:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.496057:0:98900:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98900:1874531134045056:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.496101:0:98900:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x533:0x0](ffff9d85b3ce1988) 00000080:00200000:2.0:1787695651.505323:0:98901:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.505350:0:98901:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.505415:0:98901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98901 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.505427:0:98901:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98901 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.505437:0:98901:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98901 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.505476:0:98901:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98901:1874531134045184:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.505635:0:98901:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.513510:0:98901:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98901:1874531134045184:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695651.513539:0:98901:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513552:0:98901:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513563:0:98901:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513573:0:98901:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513585:0:98901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513592:0:98901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513604:0:98901:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695651.513616:0:98901:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1180 x1874531134045184/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695651.513643:0:98901:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513655:0:98901:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695651.513661:0:98901:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695651.513688:0:98901:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:0.0:1787695651.513694:0:98901:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f45 00000080:00010000:0.0:1787695651.513699:0:98901:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695651.513702:0:98901:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f45 from it ffffb5d40415bbe0 00010000:00010000:0.0:1787695651.513706:0:98901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.513715:0:98901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.513738:0:98901:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695651.513763:0:98901:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695651.513768:0:98901:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695651.513783:0:98901:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.513793:0:98901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.513808:0:98901:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.513813:0:98901:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695651.513818:0:98901:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695651.513820:0:98901:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f45 00000080:00010000:0.0:1787695651.513824:0:98901:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f45 from it ffffb5d40415bce0 00010000:00010000:0.0:1787695651.513827:0:98901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.513835:0:98901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695651.513843:0:98901:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695651.513876:0:98901:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695651.513883:0:98901:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.513903:0:98901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.513912:0:98901:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695651.513919:0:98901:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695651.513924:0:98901:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695651.513926:0:98901:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f45 00000080:00010000:0.0:1787695651.513930:0:98901:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f45 from it ffffb5d40415bdc8 00010000:00010000:0.0:1787695651.513934:0:98901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.513941:0:98901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695651.513950:0:98901:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695651.513955:0:98901:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695651.514138:0:98901:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695651.514183:0:98901:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695651.514190:0:98901:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695651.514192:0:98901:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95328f45 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee52b expref: -99 pid: 98901 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695651.514218:0:98901:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695651.514226:0:98901:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f39500 x1874531134045440/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695651.514248:0:98901:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98901:1874531134045440:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695651.514338:0:98901:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1600 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.519417:0:98901:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98901:1874531134045440:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695651.519478:0:98901:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x534:0x0](ffff9d85b3cd3248) 00000080:00200000:0.0:1787695651.519485:0:98901:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x534:0x0], updating i_size 512 00000100:00100000:2.0:1787695651.531617:0:98902:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98902:1874531134045568:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.531664:0:98902:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531679:0:98902:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531692:0:98902:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531706:0:98902:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531719:0:98902:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531729:0:98902:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531736:0:98902:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.531749:0:98902:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2d500 x1874531134045568/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.531766:0:98902:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531784:0:98902:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.531791:0:98902:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.531830:0:98902:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.531837:0:98902:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328f4c 00000080:00010000:2.0:1787695651.531844:0:98902:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.531846:0:98902:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f4c from it ffffb5d404183c20 00010000:00010000:2.0:1787695651.531882:0:98902:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.531892:0:98902:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.531933:0:98902:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.531962:0:98902:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695651.531967:0:98902:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695651.532005:0:98902:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695651.532056:0:98902:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98902 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.532064:0:98902:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98902 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.532074:0:98902:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98902 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.532099:0:98902:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98902:1874531134045696:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.532170:0:98902:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.536789:0:98902:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98902:1874531134045696:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.536821:0:98902:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.536848:0:98902:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x534:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.536889:0:98902:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.536910:0:98902:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.536923:0:98902:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.536940:0:98902:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.536949:0:98902:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.536956:0:98902:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.536968:0:98902:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1f80 x1874531134045696/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.537494:0:98902:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.537507:0:98902:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1332 (0 0) 00000080:00200000:2.0:1787695651.537514:0:98902:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695651.537547:0:98902:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x534:0x0](ffff9d85b3cf0908) 00000080:00200000:2.0:1787695651.537556:0:98902:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x534:0x0], updating i_size 512 00000080:00010000:2.0:1787695651.537563:0:98902:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x534:0x0](ffff9d85b3cf0908) for lock 0x678ae32e95328f53 00000080:00010000:2.0:1787695651.537573:0:98902:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x534:0x0](ffff9d85b3cf0908) 00000080:00010000:2.0:1787695651.537577:0:98902:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f53 from it ffffb5d404183e08 00010000:00010000:2.0:1787695651.537580:0:98902:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.537664:0:98902:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.537704:0:98902:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.537724:0:98902:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.537738:0:98902:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.537749:0:98902:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.537756:0:98902:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.537761:0:98902:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.537764:0:98902:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328f4c 00000080:00010000:2.0:1787695651.537768:0:98902:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f4c from it ffffb5d404183d90 00010000:00010000:2.0:1787695651.537771:0:98902:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.537779:0:98902:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.537789:0:98902:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.537798:0:98902:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695651.537820:0:98902:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.538202:0:98902:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.538239:0:98902:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.538246:0:98902:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.538256:0:98902:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695651.538258:0:98902:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f53 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x534:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee563 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.538271:0:98902:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030c00/0x678ae32e95328f4c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee547 expref: -99 pid: 98902 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695651.538285:0:98902:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.538294:0:98902:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce1f80 x1874531134045952/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.538316:0:98902:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98902:1874531134045952:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695651.538393:0:98902:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.543597:0:98902:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98902:1874531134045952:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.543652:0:98902:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x534:0x0](ffff9d85b3cf0908) 00000080:00200000:2.0:1787695651.555098:0:98903:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.555122:0:98903:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.555183:0:98903:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98903 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.555193:0:98903:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98903 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.555202:0:98903:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98903 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.555228:0:98903:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98903:1874531134046080:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695651.555376:0:98903:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.559227:0:98903:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98903:1874531134046080:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695651.559255:0:98903:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559268:0:98903:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559278:0:98903:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559289:0:98903:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559301:0:98903:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559309:0:98903:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559316:0:98903:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.559327:0:98903:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1180 x1874531134046080/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.559341:0:98903:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559359:0:98903:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.559365:0:98903:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.559393:0:98903:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.559400:0:98903:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f5a 00000080:00010000:2.0:1787695651.559405:0:98903:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.559408:0:98903:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f5a from it ffffb5d40418bbe0 00010000:00010000:2.0:1787695651.559412:0:98903:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.559419:0:98903:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.559456:0:98903:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.559483:0:98903:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695651.559488:0:98903:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.559501:0:98903:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.559510:0:98903:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.559521:0:98903:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.559527:0:98903:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.559532:0:98903:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.559535:0:98903:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f5a 00000080:00010000:2.0:1787695651.559538:0:98903:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f5a from it ffffb5d40418bce0 00010000:00010000:2.0:1787695651.559542:0:98903:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.559549:0:98903:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.559557:0:98903:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.559566:0:98903:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695651.559572:0:98903:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.559578:0:98903:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.559877:0:98903:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.559885:0:98903:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.559890:0:98903:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695651.559893:0:98903:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95328f5a 00000080:00010000:2.0:1787695651.559907:0:98903:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f5a from it ffffb5d40418bdc8 00010000:00010000:2.0:1787695651.559910:0:98903:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.559917:0:98903:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695651.559927:0:98903:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.559932:0:98903:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695651.559963:0:98903:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.560120:0:98903:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695651.560127:0:98903:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695651.560130:0:98903:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95328f5a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee586 expref: -99 pid: 98903 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695651.560154:0:98903:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.560161:0:98903:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce1180 x1874531134046336/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.560181:0:98903:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98903:1874531134046336:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695651.560312:0:98903:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.567104:0:98903:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98903:1874531134046336:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695651.567161:0:98903:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x535:0x0](ffff9d85b3cf0908) 00000080:00200000:2.0:1787695651.567169:0:98903:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x535:0x0], updating i_size 512 00000080:00200000:0.0:1787695651.612258:0:98905:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695651.612286:0:98905:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.612359:0:98905:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e95328f6f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.612371:0:98905:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e95328f6f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.612381:0:98905:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e95328f6f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98905 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695651.612408:0:98905:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98905:1874531134046976:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695651.612525:0:98905:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1600 going to sleep for 12 seconds 00000080:00200000:2.0:1787695651.642516:0:98906:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.642619:0:98906:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.642635:0:98906:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695651.642690:0:98906:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98906 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.642700:0:98906:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98906 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.642710:0:98906:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98906 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.642739:0:98906:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98906:1874531134047360:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.643139:0:98906:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.651412:0:98906:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98906:1874531134047360:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.651452:0:98906:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651465:0:98906:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651477:0:98906:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651489:0:98906:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651502:0:98906:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651510:0:98906:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651517:0:98906:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.651531:0:98906:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce2300 x1874531134047360/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.651562:0:98906:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651575:0:98906:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695651.651582:0:98906:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695651.651610:0:98906:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695651.651616:0:98906:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328f76 00000080:00010000:2.0:1787695651.651621:0:98906:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.651635:0:98906:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f76 from it ffffb5d40413bc20 00010000:00010000:2.0:1787695651.651639:0:98906:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651648:0:98906:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.651671:0:98906:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695651.651693:0:98906:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695651.651698:0:98906:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695651.651720:0:98906:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695651.651763:0:98906:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98906 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651770:0:98906:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98906 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.651778:0:98906:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98906 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.651802:0:98906:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98906:1874531134047488:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.651911:0:98906:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.659260:0:98906:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98906:1874531134047488:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695651.659294:0:98906:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659320:0:98906:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x536:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659328:0:98906:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659338:0:98906:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659350:0:98906:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659362:0:98906:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659370:0:98906:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659376:0:98906:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695651.659392:0:98906:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce2300 x1874531134047488/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695651.659405:0:98906:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659414:0:98906:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1334 (0 0) 00000080:00200000:2.0:1787695651.659422:0:98906:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695651.659468:0:98906:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x536:0x0](ffff9d85b3cf1148) 00000080:00200000:2.0:1787695651.659476:0:98906:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x536:0x0], updating i_size 512 00000080:00010000:2.0:1787695651.659485:0:98906:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x536:0x0](ffff9d85b3cf1148) for lock 0x678ae32e95328f7d 00000080:00010000:2.0:1787695651.659498:0:98906:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x536:0x0](ffff9d85b3cf1148) 00000080:00010000:2.0:1787695651.659502:0:98906:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f7d from it ffffb5d40413be08 00010000:00010000:2.0:1787695651.659505:0:98906:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.659513:0:98906:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.659561:0:98906:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.659580:0:98906:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.659590:0:98906:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.659602:0:98906:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695651.659613:0:98906:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695651.659619:0:98906:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695651.659622:0:98906:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328f76 00000080:00010000:2.0:1787695651.659626:0:98906:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328f76 from it ffffb5d40413bd90 00010000:00010000:2.0:1787695651.659630:0:98906:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.659637:0:98906:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695651.659646:0:98906:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695651.659654:0:98906:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695651.659677:0:98906:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.659713:0:98906:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.659749:0:98906:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 1/0,0 mode: --/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.659757:0:98906:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.659762:0:98906:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695651.659765:0:98906:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328f7d lrc: 0/0,0 mode: --/PR res: [0x200000401:0x536:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee619 expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.659787:0:98906:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328f76 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee5fd expref: -99 pid: 98906 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695651.659802:0:98906:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.659809:0:98906:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9ce2300 x1874531134047744/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.659831:0:98906:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98906:1874531134047744:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695651.659955:0:98906:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.669289:0:98906:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98906:1874531134047744:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.669336:0:98906:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x536:0x0](ffff9d85b3cf1148) 00000080:00200000:2.0:1787695651.766360:0:98910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695651.766393:0:98910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695651.766411:0:98910:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695651.766496:0:98910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98910 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.766506:0:98910:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98910 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695651.766515:0:98910:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98910 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695651.766540:0:98910:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98910:1874531134049152:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695651.766802:0:98910:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.771110:0:98910:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98910:1874531134049152:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695651.771138:0:98910:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771150:0:98910:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771162:0:98910:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771172:0:98910:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771184:0:98910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771192:0:98910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771200:0:98910:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695651.771212:0:98910:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55c00 x1874531134049152/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695651.771226:0:98910:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771239:0:98910:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695651.771246:0:98910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695651.771272:0:98910:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:0.0:1787695651.771278:0:98910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328fa0 00000080:00010000:0.0:1787695651.771283:0:98910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695651.771286:0:98910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328fa0 from it ffffb5d40415bc20 00010000:00010000:0.0:1787695651.771289:0:98910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771298:0:98910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.771321:0:98910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695651.771342:0:98910:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695651.771347:0:98910:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695651.771369:0:98910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695651.771413:0:98910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98910 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771421:0:98910:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98910 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.771431:0:98910:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98910 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695651.771463:0:98910:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98910:1874531134049280:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695651.771549:0:98910:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:0.0:1787695651.774789:0:98910:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98910:1874531134049280:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695651.774820:0:98910:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774841:0:98910:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x538:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774868:0:98910:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774881:0:98910:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774898:0:98910:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774910:0:98910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774924:0:98910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774932:0:98910:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695651.774943:0:98910:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3b800 x1874531134049280/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695668 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695651.774955:0:98910:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.774964:0:98910:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1336 (0 0) 00000080:00200000:0.0:1787695651.774976:0:98910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695651.775002:0:98910:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x538:0x0](ffff9d85b3cd42c8) 00000080:00200000:0.0:1787695651.775009:0:98910:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x538:0x0], updating i_size 512 00000080:00010000:0.0:1787695651.775015:0:98910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x538:0x0](ffff9d85b3cd42c8) for lock 0x678ae32e95328fa7 00000080:00010000:0.0:1787695651.775024:0:98910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x538:0x0](ffff9d85b3cd42c8) 00000080:00010000:0.0:1787695651.775028:0:98910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328fa7 from it ffffb5d40415be08 00010000:00010000:0.0:1787695651.775031:0:98910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695651.775040:0:98910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.775070:0:98910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695651.775084:0:98910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695651.775094:0:98910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.775104:0:98910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695651.775111:0:98910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695651.775116:0:98910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695651.775119:0:98910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328fa0 00000080:00010000:0.0:1787695651.775122:0:98910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328fa0 from it ffffb5d40415bd90 00010000:00010000:0.0:1787695651.775125:0:98910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.775131:0:98910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695651.775140:0:98910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695651.775147:0:98910:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695651.775168:0:98910:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.775200:0:98910:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.775234:0:98910:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.775242:0:98910:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695651.775252:0:98910:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695651.775255:0:98910:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95328fa7 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x538:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee6cf expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695651.775272:0:98910:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328fa0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee6b3 expref: -99 pid: 98910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695651.775288:0:98910:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695651.775294:0:98910:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3b800 x1874531134049536/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695651.775316:0:98910:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98910:1874531134049536:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695651.775378:0:98910:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.780375:0:98910:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98910:1874531134049536:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.780425:0:98910:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x538:0x0](ffff9d85b3cd42c8) 00000080:00200000:2.0:1787695651.834045:0:98912:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695651.834167:0:98912:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328fb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee70e expref: -99 pid: 98912 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.834216:0:98912:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e000/0x678ae32e95328fbc lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x539:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee72a expref: -99 pid: 98912 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.834262:0:98912:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e000/0x678ae32e95328fbc lrc: 1/0,0 mode: --/PR res: [0x200000401:0x539:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee72a expref: -99 pid: 98912 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.834270:0:98912:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328fb5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee70e expref: -99 pid: 98912 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695651.834276:0:98912:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695651.834280:0:98912:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e000/0x678ae32e95328fbc lrc: 0/0,0 mode: --/PR res: [0x200000401:0x539:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee72a expref: -99 pid: 98912 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695651.834300:0:98912:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f200/0x678ae32e95328fb5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee70e expref: -99 pid: 98912 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695651.834316:0:98912:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695651.834329:0:98912:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f54e00 x1874531134050432/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695651.834354:0:98912:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98912:1874531134050432:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695651.834524:0:98912:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695651.840792:0:98912:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98912:1874531134050432:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695651.840843:0:98912:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x539:0x0](ffff9d85b3ceb248) 00000080:00200000:2.0:1787695651.932638:0:98913:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x53a:0x0](ffff9d85b3cf1148) 00000080:00200000:2.0:1787695651.932654:0:98913:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x53a:0x0], updating i_size 512 00000100:00100000:2.0:1787695651.995530:0:98915:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98915:1874531134051712:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695651.995636:0:98915:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.004521:0:98915:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fdf80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98915:1874531134051712:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.004572:0:98915:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x53b:0x0](ffff9d85b3cf7448) 00000080:00200000:2.0:1787695652.004591:0:98915:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x53b:0x0], updating i_size 512 00000080:00200000:2.0:1787695652.019661:0:98916:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.019709:0:98916:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.019725:0:98916:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695652.019779:0:98916:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328fdf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.019789:0:98916:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328fdf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.019799:0:98916:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328fdf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98916 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.019827:0:98916:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f57480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98916:1874531134051840:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.051135:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695652.102733:0:98917:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fdc00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98917:1874531134052608:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.102837:0:98917:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.110586:0:98917:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fdc00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98917:1874531134052608:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.110645:0:98917:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x53c:0x0](ffff9d85b3cf3a88) 00000080:00200000:2.0:1787695652.110654:0:98917:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x53c:0x0], updating i_size 512 00000080:00200000:2.0:1787695652.118249:0:98918:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.118277:0:98918:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.118293:0:98918:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695652.118345:0:98918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98918 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.118354:0:98918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98918 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.118364:0:98918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98918 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.118391:0:98918:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98918:1874531134052736:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.119162:0:98918:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000080:00200000:2.0:1787695652.131351:0:98918:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.131359:0:98918:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ff4 00000080:00010000:2.0:1787695652.131365:0:98918:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.131368:0:98918:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ff4 from it ffffb5d4041d3c20 00010000:00010000:2.0:1787695652.131372:0:98918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.131381:0:98918:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.131406:0:98918:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.131441:0:98918:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695652.131446:0:98918:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695652.131467:0:98918:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695652.131509:0:98918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98918 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.131517:0:98918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98918 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.131526:0:98918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98918 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.131544:0:98918:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98918:1874531134052864:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.131686:0:98918:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.142369:0:98918:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98918:1874531134052864:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.142405:0:98918:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142430:0:98918:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142453:0:98918:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142464:0:98918:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142474:0:98918:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142487:0:98918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142501:0:98918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142508:0:98918:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.142519:0:98918:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55f80 x1874531134052864/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.142532:0:98918:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142542:0:98918:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1340 (0 0) 00000080:00200000:2.0:1787695652.142550:0:98918:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695652.142586:0:98918:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x53c:0x0](ffff9d85b3cf1988) 00000080:00200000:2.0:1787695652.142594:0:98918:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x53c:0x0], updating i_size 512 00000080:00010000:2.0:1787695652.142601:0:98918:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x53c:0x0](ffff9d85b3cf1988) for lock 0x678ae32e95328ffb 00000080:00010000:2.0:1787695652.142611:0:98918:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x53c:0x0](ffff9d85b3cf1988) 00000080:00010000:2.0:1787695652.142615:0:98918:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ffb from it ffffb5d4041d3e08 00010000:00010000:2.0:1787695652.142618:0:98918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.142627:0:98918:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.142659:0:98918:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.142806:0:98918:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.142819:0:98918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.142831:0:98918:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.142836:0:98918:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.142842:0:98918:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.142845:0:98918:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95328ff4 00000080:00010000:2.0:1787695652.143576:0:98918:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95328ff4 from it ffffb5d4041d3d90 00010000:00010000:2.0:1787695652.143582:0:98918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.143633:0:98918:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.143643:0:98918:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.143650:0:98918:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695652.143816:0:98918:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.143846:0:98918:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.146175:0:98918:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 1/0,0 mode: --/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.146569:0:98918:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.146652:0:98918:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695652.146656:0:98918:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95328ffb lrc: 0/0,0 mode: --/PR res: [0x200000401:0x53c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee83b expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.146798:0:98918:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95328ff4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee81f expref: -99 pid: 98918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695652.146819:0:98918:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.146831:0:98918:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f55f80 x1874531134053120/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.147066:0:98918:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98918:1874531134053120:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695652.147161:0:98918:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.158536:0:98918:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98918:1874531134053120:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695652.158600:0:98918:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x53c:0x0](ffff9d85b3cf1988) 00000080:00200000:2.0:1787695652.257814:0:98921:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.257842:0:98921:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.258221:0:98921:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98921 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.258232:0:98921:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98921 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.258244:0:98921:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98921 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.258272:0:98921:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98921:1874531134054144:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695652.258426:0:98921:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.263682:0:98921:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98921:1874531134054144:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.263712:0:98921:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263727:0:98921:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263740:0:98921:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263753:0:98921:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263767:0:98921:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263776:0:98921:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263784:0:98921:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.263797:0:98921:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f54000 x1874531134054144/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.263811:0:98921:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263831:0:98921:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.263838:0:98921:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.263908:0:98921:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.263915:0:98921:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329017 00000080:00010000:2.0:1787695652.263921:0:98921:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.263925:0:98921:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329017 from it ffffb5d40414bbe0 00010000:00010000:2.0:1787695652.263928:0:98921:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.263937:0:98921:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.263973:0:98921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.264000:0:98921:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.264005:0:98921:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.264020:0:98921:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.264029:0:98921:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.264041:0:98921:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.264048:0:98921:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.264053:0:98921:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.264056:0:98921:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329017 00000080:00010000:2.0:1787695652.264059:0:98921:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329017 from it ffffb5d40414bce0 00010000:00010000:2.0:1787695652.264062:0:98921:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.264069:0:98921:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.264078:0:98921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.264084:0:98921:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.264091:0:98921:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.264096:0:98921:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.264104:0:98921:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.264111:0:98921:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.264114:0:98921:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.264117:0:98921:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329017 00000080:00010000:2.0:1787695652.264120:0:98921:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329017 from it ffffb5d40414bdc8 00010000:00010000:2.0:1787695652.264123:0:98921:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.264131:0:98921:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.264140:0:98921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.264147:0:98921:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.264319:0:98921:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.264370:0:98921:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.264378:0:98921:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.264380:0:98921:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329017 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee8b9 expref: -99 pid: 98921 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.264406:0:98921:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.264414:0:98921:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f54000 x1874531134054400/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.264447:0:98921:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98921:1874531134054400:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.264525:0:98921:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.273249:0:98921:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98921:1874531134054400:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.273308:0:98921:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x53e:0x0](ffff9d85b3cf1988) 00000080:00200000:2.0:1787695652.273316:0:98921:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x53e:0x0], updating i_size 512 00000080:00200000:2.0:1787695652.286401:0:98922:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.286633:0:98922:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.286649:0:98922:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695652.286708:0:98922:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98922 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.286718:0:98922:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98922 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.286729:0:98922:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98922 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.286759:0:98922:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98922:1874531134054528:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.287090:0:98922:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.291405:0:98922:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98922:1874531134054528:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.291445:0:98922:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291458:0:98922:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291472:0:98922:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291483:0:98922:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291496:0:98922:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291506:0:98922:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291514:0:98922:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.291526:0:98922:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f54380 x1874531134054528/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.291547:0:98922:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291560:0:98922:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.291566:0:98922:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.291593:0:98922:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.291599:0:98922:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532901e 00000080:00010000:2.0:1787695652.291604:0:98922:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.291608:0:98922:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532901e from it ffffb5d40413bc20 00010000:00010000:2.0:1787695652.291611:0:98922:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291621:0:98922:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.291644:0:98922:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.291665:0:98922:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695652.291670:0:98922:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695652.291688:0:98922:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695652.291725:0:98922:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98922 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291733:0:98922:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98922 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.291743:0:98922:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98922 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.291768:0:98922:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98922:1874531134054656:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.291836:0:98922:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.295203:0:98922:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98922:1874531134054656:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.295231:0:98922:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295253:0:98922:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295262:0:98922:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295274:0:98922:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295285:0:98922:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295298:0:98922:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295306:0:98922:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295314:0:98922:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.295331:0:98922:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f54380 x1874531134054656/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.295344:0:98922:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295355:0:98922:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1342 (0 0) 00000080:00200000:2.0:1787695652.295361:0:98922:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695652.295393:0:98922:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x53e:0x0](ffff9d85b3cf4b08) 00000080:00200000:2.0:1787695652.295401:0:98922:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x53e:0x0], updating i_size 512 00000080:00010000:2.0:1787695652.295408:0:98922:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x53e:0x0](ffff9d85b3cf4b08) for lock 0x678ae32e95329025 00000080:00010000:2.0:1787695652.295418:0:98922:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x53e:0x0](ffff9d85b3cf4b08) 00000080:00010000:2.0:1787695652.295421:0:98922:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329025 from it ffffb5d40413be08 00010000:00010000:2.0:1787695652.295425:0:98922:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.295447:0:98922:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.295479:0:98922:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.295494:0:98922:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.295504:0:98922:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.295515:0:98922:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.295528:0:98922:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.295532:0:98922:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.295536:0:98922:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532901e 00000080:00010000:2.0:1787695652.295539:0:98922:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532901e from it ffffb5d40413bd90 00010000:00010000:2.0:1787695652.295543:0:98922:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.295551:0:98922:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.296686:0:98922:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.296696:0:98922:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695652.296729:0:98922:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.296771:0:98922:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.296814:0:98922:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.296823:0:98922:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.296829:0:98922:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695652.296832:0:98922:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329025 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x53e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee8f1 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.296896:0:98922:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532901e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee8d5 expref: -99 pid: 98922 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695652.296914:0:98922:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.296925:0:98922:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f54380 x1874531134054912/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.296950:0:98922:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98922:1874531134054912:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695652.297034:0:98922:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.301914:0:98922:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98922:1874531134054912:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695652.301956:0:98922:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x53e:0x0](ffff9d85b3cf4b08) 00000080:00200000:2.0:1787695652.313334:0:98923:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.313361:0:98923:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.313424:0:98923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.313446:0:98923:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.313457:0:98923:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98923 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.313485:0:98923:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98923:1874531134055040:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695652.314128:0:98923:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.317729:0:98923:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98923:1874531134055040:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.317757:0:98923:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.317773:0:98923:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.317786:0:98923:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.317799:0:98923:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.317812:0:98923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.317822:0:98923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.317831:0:98923:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.317842:0:98923:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55180 x1874531134055040/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.318038:0:98923:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.318052:0:98923:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.318059:0:98923:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.318086:0:98923:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.318093:0:98923:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532902c 00000080:00010000:2.0:1787695652.318098:0:98923:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.318101:0:98923:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532902c from it ffffb5d40412bbe0 00010000:00010000:2.0:1787695652.318104:0:98923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.318113:0:98923:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.318136:0:98923:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.318158:0:98923:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.318162:0:98923:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.318176:0:98923:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.318184:0:98923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.318196:0:98923:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.318203:0:98923:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.318207:0:98923:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.318209:0:98923:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532902c 00000080:00010000:2.0:1787695652.318212:0:98923:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532902c from it ffffb5d40412bce0 00010000:00010000:2.0:1787695652.318215:0:98923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.318223:0:98923:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.318237:0:98923:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.318243:0:98923:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.318249:0:98923:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.318254:0:98923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.318261:0:98923:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.318266:0:98923:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.318270:0:98923:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.318273:0:98923:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532902c 00000080:00010000:2.0:1787695652.318276:0:98923:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532902c from it ffffb5d40412bdc8 00010000:00010000:2.0:1787695652.318278:0:98923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.318285:0:98923:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.318292:0:98923:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.318297:0:98923:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.318327:0:98923:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.318375:0:98923:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.318387:0:98923:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.318390:0:98923:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532902c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee914 expref: -99 pid: 98923 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.318417:0:98923:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.318424:0:98923:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f55180 x1874531134055296/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.318459:0:98923:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98923:1874531134055296:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.318741:0:98923:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.324070:0:98923:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98923:1874531134055296:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.324117:0:98923:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x53f:0x0](ffff9d85b3cf5348) 00000080:00200000:2.0:1787695652.324125:0:98923:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x53f:0x0], updating i_size 512 00000080:00200000:0.0:1787695652.355528:0:98924:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x53f:0x0](ffff9d85b3ce5b88) 00000100:00100000:2.0:1787695652.369336:0:98925:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98925:1874531134055936:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.369362:0:98925:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369375:0:98925:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369387:0:98925:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369399:0:98925:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369418:0:98925:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369438:0:98925:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369446:0:98925:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.369459:0:98925:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2c700 x1874531134055936/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.369471:0:98925:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369483:0:98925:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.369489:0:98925:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.369515:0:98925:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.369521:0:98925:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329041 00000080:00010000:2.0:1787695652.369526:0:98925:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.369529:0:98925:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329041 from it ffffb5d4041ebbe0 00010000:00010000:2.0:1787695652.369532:0:98925:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.369541:0:98925:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.369567:0:98925:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.369589:0:98925:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.369594:0:98925:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.369608:0:98925:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.369618:0:98925:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.369700:0:98925:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.369707:0:98925:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.369711:0:98925:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.369713:0:98925:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329041 00000080:00010000:2.0:1787695652.369717:0:98925:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329041 from it ffffb5d4041ebce0 00010000:00010000:2.0:1787695652.369719:0:98925:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.369726:0:98925:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.369733:0:98925:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.369739:0:98925:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.369745:0:98925:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.369751:0:98925:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.369762:0:98925:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.369768:0:98925:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.369771:0:98925:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.369773:0:98925:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329041 00000080:00010000:2.0:1787695652.369777:0:98925:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329041 from it ffffb5d4041ebdc8 00010000:00010000:2.0:1787695652.369779:0:98925:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.369786:0:98925:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.369792:0:98925:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.369797:0:98925:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.369825:0:98925:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.370330:0:98925:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.370339:0:98925:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.370342:0:98925:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030200/0x678ae32e95329041 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee96f expref: -99 pid: 98925 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.370367:0:98925:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.370375:0:98925:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f54a80 x1874531134056192/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.370397:0:98925:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98925:1874531134056192:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.370529:0:98925:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.374833:0:98925:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98925:1874531134056192:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.374927:0:98925:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x540:0x0](ffff9d85b3cf4b08) 00000080:00200000:2.0:1787695652.374936:0:98925:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x540:0x0], updating i_size 512 00000080:00200000:2.0:1787695652.384461:0:98926:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.384491:0:98926:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.384507:0:98926:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695652.384563:0:98926:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329048 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98926 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.384574:0:98926:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329048 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98926 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.384585:0:98926:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329048 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98926 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.384610:0:98926:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98926:1874531134056320:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.384719:0:98926:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000080:00200000:0.0:1787695652.455709:0:98928:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695652.455718:0:98928:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532905d 00000080:00010000:0.0:1787695652.455726:0:98928:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.455729:0:98928:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532905d from it ffffb5d40414bc20 00010000:00010000:0.0:1787695652.455734:0:98928:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.455743:0:98928:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.455771:0:98928:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695652.455800:0:98928:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695652.455805:0:98928:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695652.455829:0:98928:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695652.455918:0:98928:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98928 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.455926:0:98928:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98928 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.455936:0:98928:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98928 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.455967:0:98928:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98928:1874531134057344:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.456053:0:98928:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.460064:0:98928:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98928:1874531134057344:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.460102:0:98928:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460125:0:98928:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x541:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460140:0:98928:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460152:0:98928:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460163:0:98928:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460176:0:98928:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460186:0:98928:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460195:0:98928:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.460207:0:98928:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39500 x1874531134057344/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.460220:0:98928:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460230:0:98928:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1345 (0 0) 00000080:00200000:2.0:1787695652.460237:0:98928:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695652.460269:0:98928:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x541:0x0](ffff9d85b3cf5b88) 00000080:00200000:2.0:1787695652.460277:0:98928:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x541:0x0], updating i_size 512 00000080:00010000:2.0:1787695652.460283:0:98928:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x541:0x0](ffff9d85b3cf5b88) for lock 0x678ae32e95329064 00000080:00010000:2.0:1787695652.460293:0:98928:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x541:0x0](ffff9d85b3cf5b88) 00000080:00010000:2.0:1787695652.460297:0:98928:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329064 from it ffffb5d40414be08 00010000:00010000:2.0:1787695652.460300:0:98928:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.460309:0:98928:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.460340:0:98928:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.460355:0:98928:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.460366:0:98928:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.460377:0:98928:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.460385:0:98928:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.460389:0:98928:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.460392:0:98928:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532905d 00000080:00010000:2.0:1787695652.460396:0:98928:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532905d from it ffffb5d40414bd90 00010000:00010000:2.0:1787695652.460400:0:98928:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.460408:0:98928:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.460417:0:98928:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.460424:0:98928:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695652.460457:0:98928:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.460490:0:98928:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.460522:0:98928:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.460530:0:98928:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.460537:0:98928:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695652.460540:0:98928:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d600/0x678ae32e95329064 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x541:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea02 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.460556:0:98928:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532905d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ee9e6 expref: -99 pid: 98928 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695652.460572:0:98928:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.460579:0:98928:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f57800 x1874531134057600/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.460602:0:98928:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f57800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98928:1874531134057600:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695652.460670:0:98928:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.466234:0:98928:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f57800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98928:1874531134057600:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695652.466282:0:98928:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x541:0x0](ffff9d85b3cf5b88) 00000080:00200000:0.0:1787695652.476542:0:98929:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.476574:0:98929:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.476656:0:98929:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98929 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.476668:0:98929:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98929 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.476680:0:98929:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98929 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.476715:0:98929:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98929:1874531134057728:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695652.477099:0:98929:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.480350:0:98929:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98929:1874531134057728:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695652.480377:0:98929:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480391:0:98929:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480405:0:98929:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480417:0:98929:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480441:0:98929:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480450:0:98929:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480458:0:98929:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.480471:0:98929:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f38e00 x1874531134057728/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.480484:0:98929:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480497:0:98929:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695652.480504:0:98929:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695652.480529:0:98929:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695652.480536:0:98929:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532906b 00000080:00010000:0.0:1787695652.480541:0:98929:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695652.480544:0:98929:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532906b from it ffffb5d4041e3be0 00010000:00010000:0.0:1787695652.480548:0:98929:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.480557:0:98929:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.480582:0:98929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695652.480609:0:98929:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695652.480614:0:98929:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.480694:0:98929:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.480705:0:98929:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.480716:0:98929:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.480723:0:98929:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.480728:0:98929:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695652.480731:0:98929:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532906b 00000080:00010000:0.0:1787695652.480734:0:98929:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532906b from it ffffb5d4041e3ce0 00010000:00010000:0.0:1787695652.480737:0:98929:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.480745:0:98929:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.480753:0:98929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.480760:0:98929:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.480767:0:98929:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.480772:0:98929:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.480780:0:98929:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.480786:0:98929:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.480790:0:98929:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695652.480792:0:98929:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532906b 00000080:00010000:0.0:1787695652.480796:0:98929:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532906b from it ffffb5d4041e3dc8 00010000:00010000:0.0:1787695652.480799:0:98929:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.480807:0:98929:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695652.480815:0:98929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.480821:0:98929:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695652.481114:0:98929:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695652.481168:0:98929:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695652.481175:0:98929:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695652.481178:0:98929:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e9532906b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea25 expref: -99 pid: 98929 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695652.481209:0:98929:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695652.481218:0:98929:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f38e00 x1874531134057984/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695652.481239:0:98929:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98929:1874531134057984:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695652.481376:0:98929:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.486763:0:98929:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98929:1874531134057984:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695652.486818:0:98929:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x542:0x0](ffff9d85b3cd2a08) 00000080:00200000:0.0:1787695652.486825:0:98929:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x542:0x0], updating i_size 512 00000080:00200000:0.0:1787695652.498296:0:98930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.498329:0:98930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.498345:0:98930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695652.498408:0:98930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.498418:0:98930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.498441:0:98930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.498469:0:98930:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3a300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98930:1874531134058112:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.498551:0:98930:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.502121:0:98930:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3a300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98930:1874531134058112:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695652.502153:0:98930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502170:0:98930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502183:0:98930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502196:0:98930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502210:0:98930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502219:0:98930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502226:0:98930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.502238:0:98930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3a300 x1874531134058112/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.502253:0:98930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502269:0:98930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695652.502276:0:98930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695652.502307:0:98930:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695652.502315:0:98930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329072 00000080:00010000:0.0:1787695652.502321:0:98930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.502325:0:98930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329072 from it ffffb5d40414bc20 00010000:00010000:0.0:1787695652.502329:0:98930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502338:0:98930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.502363:0:98930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695652.502389:0:98930:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695652.502394:0:98930:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695652.502420:0:98930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695652.502484:0:98930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502493:0:98930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.502502:0:98930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.502521:0:98930:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3a300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98930:1874531134058240:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.502633:0:98930:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.507899:0:98930:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3a300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98930:1874531134058240:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695652.507931:0:98930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508122:0:98930:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x542:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508134:0:98930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508146:0:98930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508158:0:98930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508174:0:98930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508183:0:98930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508191:0:98930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.508204:0:98930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3a300 x1874531134058240/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.508217:0:98930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508234:0:98930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1346 (0 0) 00000080:00200000:0.0:1787695652.508241:0:98930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695652.508276:0:98930:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x542:0x0](ffff9d85b3cd63c8) 00000080:00200000:0.0:1787695652.508285:0:98930:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x542:0x0], updating i_size 512 00000080:00010000:0.0:1787695652.508292:0:98930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x542:0x0](ffff9d85b3cd63c8) for lock 0x678ae32e95329079 00000080:00010000:0.0:1787695652.508304:0:98930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x542:0x0](ffff9d85b3cd63c8) 00000080:00010000:0.0:1787695652.508308:0:98930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329079 from it ffffb5d40414be08 00010000:00010000:0.0:1787695652.508312:0:98930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.508321:0:98930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.508352:0:98930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.508368:0:98930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.508378:0:98930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.508390:0:98930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.508398:0:98930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.508403:0:98930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.508406:0:98930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329072 00000080:00010000:0.0:1787695652.508410:0:98930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329072 from it ffffb5d40414bd90 00010000:00010000:0.0:1787695652.508414:0:98930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.508422:0:98930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.508449:0:98930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.508458:0:98930:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695652.508479:0:98930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.508514:0:98930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.508548:0:98930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.508556:0:98930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.508562:0:98930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695652.508564:0:98930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898dc00/0x678ae32e95329079 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x542:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea5d expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.508579:0:98930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898cc00/0x678ae32e95329072 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eea41 expref: -99 pid: 98930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695652.508594:0:98930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695652.508600:0:98930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3a300 x1874531134058496/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695652.508625:0:98930:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3a300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98930:1874531134058496:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695652.509460:0:98930:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.513299:0:98930:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3a300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98930:1874531134058496:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695652.513352:0:98930:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x542:0x0](ffff9d85b3cd63c8) 00000080:00200000:2.0:1787695652.587635:0:98933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.587674:0:98933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.587752:0:98933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.587764:0:98933:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.587776:0:98933:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98933 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.587810:0:98933:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98933:1874531134059520:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695652.588046:0:98933:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.591196:0:98933:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98933:1874531134059520:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.591226:0:98933:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591238:0:98933:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591249:0:98933:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591261:0:98933:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591274:0:98933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591289:0:98933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591297:0:98933:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.591309:0:98933:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f56d80 x1874531134059520/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.591323:0:98933:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591334:0:98933:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.591341:0:98933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.591370:0:98933:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.591377:0:98933:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329095 00000080:00010000:2.0:1787695652.591382:0:98933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.591386:0:98933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329095 from it ffffb5d40413bbe0 00010000:00010000:2.0:1787695652.591390:0:98933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.591399:0:98933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.591437:0:98933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.591465:0:98933:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.591470:0:98933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.591485:0:98933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.591494:0:98933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.591505:0:98933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.591512:0:98933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.591517:0:98933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.591520:0:98933:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329095 00000080:00010000:2.0:1787695652.591524:0:98933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329095 from it ffffb5d40413bce0 00010000:00010000:2.0:1787695652.591527:0:98933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.591535:0:98933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.591544:0:98933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.591551:0:98933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.591557:0:98933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.591563:0:98933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.591571:0:98933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.591577:0:98933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.591580:0:98933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.591583:0:98933:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329095 00000080:00010000:2.0:1787695652.591586:0:98933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329095 from it ffffb5d40413bdc8 00010000:00010000:2.0:1787695652.591590:0:98933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.591597:0:98933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.591604:0:98933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.591609:0:98933:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.591643:0:98933:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.591685:0:98933:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.591693:0:98933:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.591696:0:98933:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329095 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeadb expref: -99 pid: 98933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.591721:0:98933:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.591729:0:98933:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f56d80 x1874531134059776/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.591753:0:98933:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98933:1874531134059776:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.591846:0:98933:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.597127:0:98933:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98933:1874531134059776:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.597177:0:98933:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x544:0x0](ffff9d85b3cf5b88) 00000080:00200000:2.0:1787695652.597184:0:98933:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x544:0x0], updating i_size 512 00000080:00200000:0.0:1787695652.604701:0:98934:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.604731:0:98934:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.604746:0:98934:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695652.604809:0:98934:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e9532909c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.604818:0:98934:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e9532909c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.604828:0:98934:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e9532909c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98934 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.604877:0:98934:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98934:1874531134059904:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.604959:0:98934:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000080:00200000:0.0:1787695652.628227:0:98935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.628253:0:98935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.628314:0:98935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98935 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.628323:0:98935:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98935 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.628335:0:98935:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98935 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.628360:0:98935:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3aa00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98935:1874531134060416:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695652.628503:0:98935:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1f00 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.631748:0:98935:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3aa00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98935:1874531134060416:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.631776:0:98935:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.631789:0:98935:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.631801:0:98935:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.631812:0:98935:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.631825:0:98935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.631835:0:98935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.631843:0:98935:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.632105:0:98935:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3aa00 x1874531134060416/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.632120:0:98935:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.632134:0:98935:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.632144:0:98935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.632178:0:98935:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.632184:0:98935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290aa 00000080:00010000:2.0:1787695652.632190:0:98935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.632193:0:98935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290aa from it ffffb5d4041e3be0 00010000:00010000:2.0:1787695652.632196:0:98935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.632205:0:98935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.632230:0:98935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.632250:0:98935:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.632255:0:98935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.632267:0:98935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.632275:0:98935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.632287:0:98935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.632294:0:98935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.632298:0:98935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.632301:0:98935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290aa 00000080:00010000:2.0:1787695652.632305:0:98935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290aa from it ffffb5d4041e3ce0 00010000:00010000:2.0:1787695652.632308:0:98935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.632322:0:98935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.632330:0:98935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.632337:0:98935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.632343:0:98935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.632349:0:98935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.632358:0:98935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.632365:0:98935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.632368:0:98935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.632371:0:98935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290aa 00000080:00010000:2.0:1787695652.632375:0:98935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290aa from it ffffb5d4041e3dc8 00010000:00010000:2.0:1787695652.632378:0:98935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.632387:0:98935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.632394:0:98935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.632400:0:98935:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.632446:0:98935:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.632497:0:98935:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.632504:0:98935:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.632508:0:98935:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e953290aa lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeb36 expref: -99 pid: 98935 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.632534:0:98935:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.632542:0:98935:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f57100 x1874531134060672/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.632564:0:98935:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f57100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98935:1874531134060672:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.632768:0:98935:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.638289:0:98935:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f57100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98935:1874531134060672:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.638349:0:98935:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x545:0x0](ffff9d85b3cf6c08) 00000080:00200000:2.0:1787695652.638358:0:98935:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x545:0x0], updating i_size 512 00000080:00200000:2.0:1787695652.645242:0:98936:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.645271:0:98936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.645287:0:98936:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695652.645344:0:98936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.645354:0:98936:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.645364:0:98936:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98936 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.645394:0:98936:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98936:1874531134060800:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.646337:0:98936:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.649399:0:98936:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98936:1874531134060800:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.649438:0:98936:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649452:0:98936:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649463:0:98936:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649473:0:98936:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649486:0:98936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649495:0:98936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649502:0:98936:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.649515:0:98936:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f56680 x1874531134060800/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.649534:0:98936:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649548:0:98936:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.649554:0:98936:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.649582:0:98936:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.649588:0:98936:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290b1 00000080:00010000:2.0:1787695652.649594:0:98936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.649598:0:98936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290b1 from it ffffb5d4041f3c20 00010000:00010000:2.0:1787695652.649601:0:98936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.649611:0:98936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.649788:0:98936:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.649812:0:98936:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695652.649818:0:98936:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695652.649835:0:98936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695652.650121:0:98936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.650130:0:98936:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.650140:0:98936:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98936 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.650167:0:98936:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98936:1874531134060928:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.650280:0:98936:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.653813:0:98936:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98936:1874531134060928:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.653838:0:98936:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.653899:0:98936:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x545:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.653909:0:98936:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.653922:0:98936:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.653933:0:98936:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.653946:0:98936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.653965:0:98936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.653973:0:98936:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.653991:0:98936:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f56680 x1874531134060928/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.654003:0:98936:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.654013:0:98936:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1349 (0 0) 00000080:00200000:2.0:1787695652.654020:0:98936:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695652.654047:0:98936:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x545:0x0](ffff9d85b3cf3248) 00000080:00200000:2.0:1787695652.654055:0:98936:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x545:0x0], updating i_size 512 00000080:00010000:2.0:1787695652.654061:0:98936:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x545:0x0](ffff9d85b3cf3248) for lock 0x678ae32e953290b8 00000080:00010000:2.0:1787695652.654072:0:98936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x545:0x0](ffff9d85b3cf3248) 00000080:00010000:2.0:1787695652.654076:0:98936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290b8 from it ffffb5d4041f3e08 00010000:00010000:2.0:1787695652.654080:0:98936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.654088:0:98936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.654119:0:98936:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.654135:0:98936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.654145:0:98936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.654157:0:98936:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.654170:0:98936:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.654175:0:98936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.654178:0:98936:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290b1 00000080:00010000:2.0:1787695652.654182:0:98936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290b1 from it ffffb5d4041f3d90 00010000:00010000:2.0:1787695652.654185:0:98936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.654194:0:98936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.654203:0:98936:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.654210:0:98936:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695652.654230:0:98936:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.654261:0:98936:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.654295:0:98936:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.654303:0:98936:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.654310:0:98936:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695652.654313:0:98936:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290b8 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x545:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeb6e expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.654334:0:98936:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290b1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeb52 expref: -99 pid: 98936 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695652.654350:0:98936:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.654356:0:98936:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f56680 x1874531134061184/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.654379:0:98936:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98936:1874531134061184:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695652.654451:0:98936:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.660214:0:98936:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98936:1874531134061184:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695652.660259:0:98936:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x545:0x0](ffff9d85b3cf3248) 00000080:00200000:2.0:1787695652.687332:0:98938:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.687363:0:98938:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.687378:0:98938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695652.687451:0:98938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.687462:0:98938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.687474:0:98938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.687501:0:98938:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98938:1874531134061696:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.687641:0:98938:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.691212:0:98938:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98938:1874531134061696:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.691239:0:98938:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691253:0:98938:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691266:0:98938:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691277:0:98938:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691289:0:98938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691362:0:98938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691371:0:98938:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.691384:0:98938:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55500 x1874531134061696/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.691398:0:98938:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691412:0:98938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.691419:0:98938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.691461:0:98938:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.691467:0:98938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290c6 00000080:00010000:2.0:1787695652.691473:0:98938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.691477:0:98938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290c6 from it ffffb5d404213c20 00010000:00010000:2.0:1787695652.691481:0:98938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691491:0:98938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.691519:0:98938:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.691548:0:98938:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695652.691553:0:98938:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695652.691574:0:98938:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695652.691614:0:98938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691621:0:98938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.691698:0:98938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.691720:0:98938:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98938:1874531134061824:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.691839:0:98938:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.695568:0:98938:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98938:1874531134061824:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.695601:0:98938:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695699:0:98938:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 4/1,0 mode: --/PR res: [0x200000401:0x546:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695708:0:98938:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 4/1,0 mode: --/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695719:0:98938:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695731:0:98938:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695746:0:98938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695754:0:98938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695761:0:98938:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.695774:0:98938:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55500 x1874531134061824/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.695787:0:98938:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.695800:0:98938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1350 (0 0) 00000080:00200000:2.0:1787695652.695808:0:98938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695652.695841:0:98938:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x546:0x0](ffff9d85b3cf3248) 00000080:00200000:2.0:1787695652.696140:0:98938:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x546:0x0], updating i_size 512 00000080:00010000:2.0:1787695652.696150:0:98938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x546:0x0](ffff9d85b3cf3248) for lock 0x678ae32e953290cd 00000080:00010000:2.0:1787695652.696160:0:98938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x546:0x0](ffff9d85b3cf3248) 00000080:00010000:2.0:1787695652.696164:0:98938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290cd from it ffffb5d404213e08 00010000:00010000:2.0:1787695652.696168:0:98938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.696178:0:98938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.696207:0:98938:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.696222:0:98938:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.696232:0:98938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.696244:0:98938:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.696251:0:98938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.696256:0:98938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.696259:0:98938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290c6 00000080:00010000:2.0:1787695652.696263:0:98938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290c6 from it ffffb5d404213d90 00010000:00010000:2.0:1787695652.696266:0:98938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.696274:0:98938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.696282:0:98938:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.696289:0:98938:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695652.696310:0:98938:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.696343:0:98938:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.696378:0:98938:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 1/0,0 mode: --/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.696386:0:98938:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.696393:0:98938:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695652.696396:0:98938:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953290cd lrc: 0/0,0 mode: --/PR res: [0x200000401:0x546:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eebc9 expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.696412:0:98938:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953290c6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eebad expref: -99 pid: 98938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695652.696441:0:98938:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.696449:0:98938:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f55500 x1874531134062080/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.696478:0:98938:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98938:1874531134062080:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695652.696559:0:98938:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.700117:0:98938:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98938:1874531134062080:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695652.700157:0:98938:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x546:0x0](ffff9d85b3cf3248) 00000080:00200000:2.0:1787695652.706406:0:98939:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.708326:0:98939:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.708408:0:98939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98939 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.708419:0:98939:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98939 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.708444:0:98939:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98939 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.708471:0:98939:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f57b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98939:1874531134062208:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695652.708606:0:98939:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.713120:0:98939:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f57b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98939:1874531134062208:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.713149:0:98939:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713162:0:98939:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713175:0:98939:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713193:0:98939:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713206:0:98939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713215:0:98939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713222:0:98939:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.713233:0:98939:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f57b80 x1874531134062208/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.713246:0:98939:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713262:0:98939:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.713269:0:98939:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.713302:0:98939:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.713308:0:98939:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290d4 00000080:00010000:2.0:1787695652.713314:0:98939:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.713317:0:98939:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290d4 from it ffffb5d40421bbe0 00010000:00010000:2.0:1787695652.713321:0:98939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.713336:0:98939:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.713362:0:98939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.713387:0:98939:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.713391:0:98939:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.713404:0:98939:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.713413:0:98939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.713435:0:98939:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.713444:0:98939:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.713449:0:98939:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.713452:0:98939:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290d4 00000080:00010000:2.0:1787695652.713456:0:98939:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290d4 from it ffffb5d40421bce0 00010000:00010000:2.0:1787695652.713460:0:98939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.713468:0:98939:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.713477:0:98939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.713483:0:98939:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.713489:0:98939:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.713495:0:98939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.713504:0:98939:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.713510:0:98939:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.713514:0:98939:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.713516:0:98939:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290d4 00000080:00010000:2.0:1787695652.713519:0:98939:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290d4 from it ffffb5d40421bdc8 00010000:00010000:2.0:1787695652.713523:0:98939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.713530:0:98939:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.713539:0:98939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.713544:0:98939:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.713574:0:98939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.713622:0:98939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.713763:0:98939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.713767:0:98939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953290d4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eebec expref: -99 pid: 98939 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.713803:0:98939:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.713811:0:98939:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f57b80 x1874531134062464/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.713833:0:98939:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f57b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98939:1874531134062464:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.714160:0:98939:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000080:00200000:0.0:1787695652.725369:0:98939:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x547:0x0](ffff9d85b3cd63c8) 00000080:00200000:0.0:1787695652.725378:0:98939:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x547:0x0], updating i_size 512 00000080:00200000:0.0:1787695652.732843:0:98940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.733032:0:98940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.733047:0:98940:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695652.733103:0:98940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.733113:0:98940:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.733124:0:98940:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98940 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.733149:0:98940:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98940:1874531134062592:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.733230:0:98940:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.736040:0:98940:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98940:1874531134062592:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695652.736067:0:98940:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736080:0:98940:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736091:0:98940:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736109:0:98940:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736121:0:98940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736130:0:98940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736139:0:98940:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.736151:0:98940:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3ad80 x1874531134062592/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.736165:0:98940:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736179:0:98940:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695652.736185:0:98940:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695652.736213:0:98940:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695652.736218:0:98940:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290db 00000080:00010000:0.0:1787695652.736224:0:98940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.736227:0:98940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290db from it ffffb5d40420bc20 00010000:00010000:0.0:1787695652.736231:0:98940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736245:0:98940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.736269:0:98940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695652.736291:0:98940:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695652.736296:0:98940:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695652.736313:0:98940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695652.736351:0:98940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736360:0:98940:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.736369:0:98940:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98940 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.736388:0:98940:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98940:1874531134062720:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.736482:0:98940:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.739465:0:98940:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98940:1874531134062720:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695652.739492:0:98940:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739513:0:98940:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x547:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739522:0:98940:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739534:0:98940:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739545:0:98940:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739559:0:98940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739568:0:98940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739576:0:98940:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.739588:0:98940:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3ad80 x1874531134062720/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.739601:0:98940:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.739611:0:98940:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1351 (0 0) 00000080:00200000:0.0:1787695652.739618:0:98940:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695652.740036:0:98940:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x547:0x0](ffff9d85b3cd21c8) 00000080:00200000:0.0:1787695652.740045:0:98940:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x547:0x0], updating i_size 512 00000080:00010000:0.0:1787695652.740053:0:98940:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x547:0x0](ffff9d85b3cd21c8) for lock 0x678ae32e953290e2 00000080:00010000:0.0:1787695652.740064:0:98940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x547:0x0](ffff9d85b3cd21c8) 00000080:00010000:0.0:1787695652.740069:0:98940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290e2 from it ffffb5d40420be08 00010000:00010000:0.0:1787695652.740073:0:98940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.740083:0:98940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.740116:0:98940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.740131:0:98940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.740142:0:98940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.740153:0:98940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.740160:0:98940:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.740164:0:98940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.740167:0:98940:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290db 00000080:00010000:0.0:1787695652.740171:0:98940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290db from it ffffb5d40420bd90 00010000:00010000:0.0:1787695652.740175:0:98940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.740183:0:98940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.740192:0:98940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.740199:0:98940:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695652.740221:0:98940:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.740255:0:98940:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.740290:0:98940:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.740298:0:98940:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.740305:0:98940:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695652.740308:0:98940:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c400/0x678ae32e953290e2 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x547:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec24 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.740326:0:98940:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e953290db lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec08 expref: -99 pid: 98940 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695652.740340:0:98940:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695652.740349:0:98940:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3ad80 x1874531134062976/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695652.740371:0:98940:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98940:1874531134062976:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695652.740881:0:98940:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.746183:0:98940:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98940:1874531134062976:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695652.746228:0:98940:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x547:0x0](ffff9d85b3cd21c8) 00000080:00200000:0.0:1787695652.755376:0:98941:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.755412:0:98941:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.755503:0:98941:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98941 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.755517:0:98941:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98941 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.755529:0:98941:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98941 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.755560:0:98941:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98941:1874531134063104:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695652.755688:0:98941:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.758528:0:98941:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98941:1874531134063104:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695652.758556:0:98941:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.758569:0:98941:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.758582:0:98941:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.758593:0:98941:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.758606:0:98941:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.758615:0:98941:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.758703:0:98941:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.758717:0:98941:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3b800 x1874531134063104/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.758732:0:98941:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.758747:0:98941:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695652.758756:0:98941:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695652.758787:0:98941:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695652.759176:0:98941:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290e9 00000080:00010000:0.0:1787695652.759184:0:98941:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695652.759187:0:98941:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290e9 from it ffffb5d404223be0 00010000:00010000:0.0:1787695652.759191:0:98941:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.759201:0:98941:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.759227:0:98941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695652.759249:0:98941:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695652.759254:0:98941:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.759268:0:98941:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.759278:0:98941:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.759294:0:98941:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.759301:0:98941:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.759306:0:98941:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695652.759309:0:98941:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290e9 00000080:00010000:0.0:1787695652.759313:0:98941:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290e9 from it ffffb5d404223ce0 00010000:00010000:0.0:1787695652.759316:0:98941:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.759325:0:98941:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.759334:0:98941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.759341:0:98941:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.759347:0:98941:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.759353:0:98941:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.759361:0:98941:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.759368:0:98941:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.759371:0:98941:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695652.759374:0:98941:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953290e9 00000080:00010000:0.0:1787695652.759377:0:98941:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290e9 from it ffffb5d404223dc8 00010000:00010000:0.0:1787695652.759381:0:98941:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.759388:0:98941:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695652.759397:0:98941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.759402:0:98941:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695652.759462:0:98941:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695652.759504:0:98941:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695652.759512:0:98941:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695652.759515:0:98941:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ce00/0x678ae32e953290e9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec47 expref: -99 pid: 98941 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695652.759543:0:98941:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695652.759551:0:98941:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3b800 x1874531134063360/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695652.759572:0:98941:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98941:1874531134063360:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695652.759745:0:98941:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.765084:0:98941:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98941:1874531134063360:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695652.765151:0:98941:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x548:0x0](ffff9d85b3cd21c8) 00000080:00200000:0.0:1787695652.765160:0:98941:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x548:0x0], updating i_size 512 00000080:00200000:0.0:1787695652.772270:0:98942:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.772299:0:98942:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.772313:0:98942:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695652.772375:0:98942:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98942 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.772385:0:98942:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98942 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.772395:0:98942:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98942 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.772435:0:98942:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98942:1874531134063488:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.772581:0:98942:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1540 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.775919:0:98942:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98942:1874531134063488:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.775957:0:98942:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.775970:0:98942:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.775982:0:98942:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.775994:0:98942:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.776007:0:98942:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.776015:0:98942:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.776022:0:98942:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.776035:0:98942:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3b100 x1874531134063488/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.776051:0:98942:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.776065:0:98942:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.776072:0:98942:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.776100:0:98942:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.776107:0:98942:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290f0 00000080:00010000:2.0:1787695652.776113:0:98942:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.776116:0:98942:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290f0 from it ffffb5d40413bc20 00010000:00010000:2.0:1787695652.776120:0:98942:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.776129:0:98942:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.776153:0:98942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.776174:0:98942:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695652.776178:0:98942:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695652.776198:0:98942:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695652.776241:0:98942:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98942 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.776248:0:98942:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98942 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.776257:0:98942:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98942 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.776280:0:98942:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98942:1874531134063616:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.776341:0:98942:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.779777:0:98942:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98942:1874531134063616:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.779806:0:98942:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779826:0:98942:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x548:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779835:0:98942:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779847:0:98942:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779898:0:98942:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779912:0:98942:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779922:0:98942:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779930:0:98942:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.779947:0:98942:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f56a00 x1874531134063616/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.779970:0:98942:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.779981:0:98942:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1352 (0 0) 00000080:00200000:2.0:1787695652.779988:0:98942:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695652.780022:0:98942:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x548:0x0](ffff9d85b3cf3248) 00000080:00200000:2.0:1787695652.780030:0:98942:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x548:0x0], updating i_size 512 00000080:00010000:2.0:1787695652.780037:0:98942:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x548:0x0](ffff9d85b3cf3248) for lock 0x678ae32e953290f7 00000080:00010000:2.0:1787695652.780049:0:98942:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x548:0x0](ffff9d85b3cf3248) 00000080:00010000:2.0:1787695652.780053:0:98942:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290f7 from it ffffb5d40413be08 00010000:00010000:2.0:1787695652.780057:0:98942:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.780066:0:98942:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.780095:0:98942:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.780109:0:98942:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.780118:0:98942:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.780128:0:98942:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.780140:0:98942:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.780145:0:98942:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.780148:0:98942:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953290f0 00000080:00010000:2.0:1787695652.780152:0:98942:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953290f0 from it ffffb5d40413bd90 00010000:00010000:2.0:1787695652.780155:0:98942:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.780163:0:98942:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.780172:0:98942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.780180:0:98942:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695652.780198:0:98942:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.780231:0:98942:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.780264:0:98942:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.780272:0:98942:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.780278:0:98942:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695652.780280:0:98942:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953290f7 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x548:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec7f expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.780302:0:98942:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e953290f0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eec63 expref: -99 pid: 98942 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695652.780317:0:98942:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.780324:0:98942:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f56a00 x1874531134063872/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.780344:0:98942:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98942:1874531134063872:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695652.780419:0:98942:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.789601:0:98942:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98942:1874531134063872:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695652.789744:0:98942:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x548:0x0](ffff9d85b3cf3248) 00000080:00200000:2.0:1787695652.840783:0:98945:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.840808:0:98945:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.841229:0:98945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98945 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.841239:0:98945:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98945 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.841250:0:98945:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98945 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.841276:0:98945:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98945:1874531134064896:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695652.841416:0:98945:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.844477:0:98945:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98945:1874531134064896:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.844506:0:98945:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844521:0:98945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844535:0:98945:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844546:0:98945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844559:0:98945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844569:0:98945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844578:0:98945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.844591:0:98945:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f54700 x1874531134064896/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.844605:0:98945:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844618:0:98945:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.844689:0:98945:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.844717:0:98945:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.844725:0:98945:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329113 00000080:00010000:2.0:1787695652.844732:0:98945:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.844735:0:98945:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329113 from it ffffb5d4041e3be0 00010000:00010000:2.0:1787695652.844740:0:98945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.844750:0:98945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.844775:0:98945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.844796:0:98945:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.844800:0:98945:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.844814:0:98945:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.844824:0:98945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.844835:0:98945:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.844842:0:98945:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.844846:0:98945:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.845178:0:98945:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329113 00000080:00010000:2.0:1787695652.845184:0:98945:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329113 from it ffffb5d4041e3ce0 00010000:00010000:2.0:1787695652.845187:0:98945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.845195:0:98945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.845210:0:98945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.845218:0:98945:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.845226:0:98945:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.845233:0:98945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.845242:0:98945:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.845249:0:98945:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.845253:0:98945:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.845256:0:98945:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329113 00000080:00010000:2.0:1787695652.845259:0:98945:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329113 from it ffffb5d4041e3dc8 00010000:00010000:2.0:1787695652.845262:0:98945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.845269:0:98945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.845277:0:98945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.845282:0:98945:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.845318:0:98945:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.845368:0:98945:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.845381:0:98945:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.845384:0:98945:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329113 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eecfd expref: -99 pid: 98945 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.845408:0:98945:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.845417:0:98945:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f54700 x1874531134065152/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.845452:0:98945:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98945:1874531134065152:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.845577:0:98945:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.850754:0:98945:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98945:1874531134065152:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.850808:0:98945:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54a:0x0](ffff9d85b3cf3248) 00000080:00200000:2.0:1787695652.850815:0:98945:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54a:0x0], updating i_size 512 00000080:00200000:2.0:1787695652.858297:0:98946:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.858325:0:98946:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.858340:0:98946:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695652.858390:0:98946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.858400:0:98946:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.858411:0:98946:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98946 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.858449:0:98946:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98946:1874531134065280:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.858577:0:98946:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.861793:0:98946:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98946:1874531134065280:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.861821:0:98946:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.861834:0:98946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.861846:0:98946:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862061:0:98946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862075:0:98946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862084:0:98946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862092:0:98946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.862105:0:98946:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f56300 x1874531134065280/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.862120:0:98946:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862134:0:98946:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.862141:0:98946:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.862167:0:98946:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.862173:0:98946:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532911a 00000080:00010000:2.0:1787695652.862179:0:98946:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.862182:0:98946:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532911a from it ffffb5d404233c20 00010000:00010000:2.0:1787695652.862186:0:98946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862195:0:98946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.862217:0:98946:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.862238:0:98946:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695652.862242:0:98946:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695652.862260:0:98946:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695652.862297:0:98946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862305:0:98946:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.862314:0:98946:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98946 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.862332:0:98946:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98946:1874531134065408:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695652.862439:0:98946:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.865807:0:98946:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98946:1874531134065408:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695652.865837:0:98946:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.865921:0:98946:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.865932:0:98946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.865943:0:98946:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.865966:0:98946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.865979:0:98946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.865988:0:98946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.865996:0:98946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.866008:0:98946:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f56300 x1874531134065408/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.866021:0:98946:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.866038:0:98946:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1354 (0 0) 00000080:00200000:2.0:1787695652.866045:0:98946:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695652.866077:0:98946:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54a:0x0](ffff9d85b3cf42c8) 00000080:00200000:2.0:1787695652.866085:0:98946:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54a:0x0], updating i_size 512 00000080:00010000:2.0:1787695652.866093:0:98946:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x54a:0x0](ffff9d85b3cf42c8) for lock 0x678ae32e95329121 00000080:00010000:2.0:1787695652.866106:0:98946:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x54a:0x0](ffff9d85b3cf42c8) 00000080:00010000:2.0:1787695652.866111:0:98946:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329121 from it ffffb5d404233e08 00010000:00010000:2.0:1787695652.866115:0:98946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.866125:0:98946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.866161:0:98946:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695652.866180:0:98946:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.866192:0:98946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.866204:0:98946:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.866212:0:98946:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.866217:0:98946:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695652.866220:0:98946:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532911a 00000080:00010000:2.0:1787695652.866224:0:98946:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532911a from it ffffb5d404233d90 00010000:00010000:2.0:1787695652.866227:0:98946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.866237:0:98946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.866247:0:98946:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.866256:0:98946:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695652.866280:0:98946:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.866322:0:98946:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.866367:0:98946:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.866376:0:98946:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.866383:0:98946:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695652.866387:0:98946:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329121 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x54a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed35 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.866407:0:98946:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532911a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed19 expref: -99 pid: 98946 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695652.866438:0:98946:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.866446:0:98946:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f56300 x1874531134065664/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.866479:0:98946:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f56300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98946:1874531134065664:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695652.866572:0:98946:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.870677:0:98946:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f56300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98946:1874531134065664:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695652.870725:0:98946:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x54a:0x0](ffff9d85b3cf42c8) 00000080:00200000:2.0:1787695652.889215:0:98947:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.889239:0:98947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.889301:0:98947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329128 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98947 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.889312:0:98947:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329128 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98947 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.889323:0:98947:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329128 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98947 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.889352:0:98947:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98947:1874531134065792:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695652.889449:0:98947:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000080:00200000:0.0:1787695652.920301:0:98948:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54b:0x0](ffff9d85b3cd00c8) 00000080:00200000:0.0:1787695652.920312:0:98948:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54b:0x0], updating i_size 512 00000080:00010000:0.0:1787695652.920321:0:98948:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x54b:0x0](ffff9d85b3cd00c8) for lock 0x678ae32e95329136 00000080:00010000:0.0:1787695652.920334:0:98948:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x54b:0x0](ffff9d85b3cd00c8) 00000080:00010000:0.0:1787695652.920339:0:98948:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329136 from it ffffb5d4041fbe08 00010000:00010000:0.0:1787695652.920343:0:98948:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e600/0x678ae32e95329136 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed90 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.920354:0:98948:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e600/0x678ae32e95329136 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed90 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.920394:0:98948:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.920411:0:98948:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.920435:0:98948:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e9532912f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed74 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.920454:0:98948:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e9532912f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed74 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.920463:0:98948:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.920468:0:98948:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.920472:0:98948:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532912f 00000080:00010000:0.0:1787695652.920476:0:98948:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532912f from it ffffb5d4041fbd90 00010000:00010000:0.0:1787695652.920480:0:98948:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e9532912f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eed74 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.920488:0:98948:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e9532912f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eed74 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.920498:0:98948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.920505:0:98948:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695652.920526:0:98948:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e9532912f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eed74 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.920561:0:98948:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e600/0x678ae32e95329136 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eed90 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.920598:0:98948:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e600/0x678ae32e95329136 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x54b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed90 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.920606:0:98948:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e9532912f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed74 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.920618:0:98948:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695652.920621:0:98948:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e600/0x678ae32e95329136 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x54b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed90 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.920709:0:98948:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e9532912f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eed74 expref: -99 pid: 98948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695652.920727:0:98948:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695652.920734:0:98948:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f39880 x1874531134066560/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695652.920760:0:98948:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98948:1874531134066560:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695652.921067:0:98948:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1600 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.926161:0:98948:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98948:1874531134066560:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695652.926212:0:98948:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x54b:0x0](ffff9d85b3cd00c8) 00000080:00200000:0.0:1787695652.937510:0:98949:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695652.937537:0:98949:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.937613:0:98949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.937625:0:98949:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.937637:0:98949:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98949 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.937663:0:98949:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3a680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98949:1874531134066688:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695652.937822:0:98949:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.941333:0:98949:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3a680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98949:1874531134066688:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.941364:0:98949:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941377:0:98949:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941389:0:98949:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941405:0:98949:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941427:0:98949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941437:0:98949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941445:0:98949:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.941457:0:98949:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3a680 x1874531134066688/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.941470:0:98949:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941482:0:98949:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.941489:0:98949:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.941518:0:98949:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.941524:0:98949:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532913d 00000080:00010000:2.0:1787695652.941530:0:98949:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.941533:0:98949:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532913d from it ffffb5d4041e3be0 00010000:00010000:2.0:1787695652.941537:0:98949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.941550:0:98949:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.941577:0:98949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.941604:0:98949:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.941609:0:98949:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.941622:0:98949:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.941693:0:98949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.941704:0:98949:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.941710:0:98949:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.941716:0:98949:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.941718:0:98949:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532913d 00000080:00010000:2.0:1787695652.941722:0:98949:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532913d from it ffffb5d4041e3ce0 00010000:00010000:2.0:1787695652.941726:0:98949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.941733:0:98949:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.941742:0:98949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.941748:0:98949:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.941755:0:98949:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.941762:0:98949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.941770:0:98949:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.941776:0:98949:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.941780:0:98949:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.941782:0:98949:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532913d 00000080:00010000:2.0:1787695652.941786:0:98949:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532913d from it ffffb5d4041e3dc8 00010000:00010000:2.0:1787695652.941789:0:98949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.941795:0:98949:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.941803:0:98949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.941808:0:98949:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.941839:0:98949:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.944334:0:98949:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.944356:0:98949:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.944360:0:98949:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532913d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eedb3 expref: -99 pid: 98949 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.944406:0:98949:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.944430:0:98949:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f54e00 x1874531134066944/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.944459:0:98949:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f54e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98949:1874531134066944:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.944593:0:98949:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.951220:0:98949:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f54e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98949:1874531134066944:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.951276:0:98949:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54c:0x0](ffff9d85b3cf42c8) 00000080:00200000:2.0:1787695652.951284:0:98949:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54c:0x0], updating i_size 512 00000080:00200000:0.0:1787695652.960258:0:98950:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.960291:0:98950:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.960306:0:98950:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695652.960367:0:98950:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98950 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.960378:0:98950:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98950 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.960389:0:98950:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98950 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.960426:0:98950:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98950:1874531134067072:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.960569:0:98950:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.963813:0:98950:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98950:1874531134067072:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695652.963842:0:98950:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964070:0:98950:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964142:0:98950:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964155:0:98950:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964176:0:98950:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964185:0:98950:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964193:0:98950:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.964206:0:98950:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f38380 x1874531134067072/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.964220:0:98950:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964235:0:98950:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695652.964243:0:98950:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695652.964270:0:98950:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695652.964277:0:98950:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329144 00000080:00010000:0.0:1787695652.964282:0:98950:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.964286:0:98950:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329144 from it ffffb5d40423bc20 00010000:00010000:0.0:1787695652.964289:0:98950:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964300:0:98950:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.964330:0:98950:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695652.964356:0:98950:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695652.964361:0:98950:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695652.964378:0:98950:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695652.964429:0:98950:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98950 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964437:0:98950:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98950 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.964447:0:98950:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98950 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695652.964470:0:98950:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98950:1874531134067200:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695652.964584:0:98950:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.969249:0:98950:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98950:1874531134067200:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695652.969280:0:98950:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969302:0:98950:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969311:0:98950:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969324:0:98950:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969337:0:98950:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969350:0:98950:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969359:0:98950:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969368:0:98950:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695652.969381:0:98950:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f38380 x1874531134067200/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695652.969394:0:98950:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969405:0:98950:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1356 (0 0) 00000080:00200000:0.0:1787695652.969413:0:98950:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695652.969457:0:98950:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54c:0x0](ffff9d85b3cd00c8) 00000080:00200000:0.0:1787695652.969466:0:98950:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54c:0x0], updating i_size 512 00000080:00010000:0.0:1787695652.969472:0:98950:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x54c:0x0](ffff9d85b3cd00c8) for lock 0x678ae32e9532914b 00000080:00010000:0.0:1787695652.969483:0:98950:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x54c:0x0](ffff9d85b3cd00c8) 00000080:00010000:0.0:1787695652.969486:0:98950:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532914b from it ffffb5d40423be08 00010000:00010000:0.0:1787695652.969490:0:98950:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695652.969504:0:98950:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.969534:0:98950:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695652.969551:0:98950:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695652.969560:0:98950:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.969571:0:98950:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695652.969577:0:98950:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695652.969580:0:98950:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695652.969583:0:98950:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329144 00000080:00010000:0.0:1787695652.969587:0:98950:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329144 from it ffffb5d40423bd90 00010000:00010000:0.0:1787695652.969590:0:98950:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.969597:0:98950:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695652.969605:0:98950:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695652.969613:0:98950:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695652.969705:0:98950:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.969742:0:98950:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.969779:0:98950:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 1/0,0 mode: --/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.969788:0:98950:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695652.969793:0:98950:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695652.969797:0:98950:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e9532914b lrc: 0/0,0 mode: --/PR res: [0x200000401:0x54c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eedeb expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695652.969811:0:98950:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c000/0x678ae32e95329144 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eedcf expref: -99 pid: 98950 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695652.969825:0:98950:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695652.969833:0:98950:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f38380 x1874531134067456/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695652.970035:0:98950:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98950:1874531134067456:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695652.970272:0:98950:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695652.975784:0:98950:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98950:1874531134067456:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695652.975833:0:98950:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x54c:0x0](ffff9d85b3cd00c8) 00000080:00200000:2.0:1787695652.986445:0:98951:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.986475:0:98951:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.986547:0:98951:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98951 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.986559:0:98951:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98951 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.986572:0:98951:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98951 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695652.986602:0:98951:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f57480 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98951:1874531134067584:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695652.986710:0:98951:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.990848:0:98951:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f57480 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98951:1874531134067584:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695652.990916:0:98951:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.990928:0:98951:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.990955:0:98951:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.990966:0:98951:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.990979:0:98951:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.990988:0:98951:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.990996:0:98951:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695652.991009:0:98951:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f57480 x1874531134067584/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695669 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695652.991024:0:98951:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.991038:0:98951:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695652.991046:0:98951:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695652.991073:0:98951:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695652.991080:0:98951:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329152 00000080:00010000:2.0:1787695652.991096:0:98951:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.991100:0:98951:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329152 from it ffffb5d404243be0 00010000:00010000:2.0:1787695652.991104:0:98951:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695652.991114:0:98951:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.991140:0:98951:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695652.991164:0:98951:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695652.991169:0:98951:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.991183:0:98951:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.991193:0:98951:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.991204:0:98951:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695652.991211:0:98951:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.991216:0:98951:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.991219:0:98951:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329152 00000080:00010000:2.0:1787695652.991222:0:98951:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329152 from it ffffb5d404243ce0 00010000:00010000:2.0:1787695652.991226:0:98951:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695652.991234:0:98951:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.991242:0:98951:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.991249:0:98951:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695652.991256:0:98951:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695652.991261:0:98951:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.991269:0:98951:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695652.991275:0:98951:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695652.991279:0:98951:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695652.991282:0:98951:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329152 00000080:00010000:2.0:1787695652.991285:0:98951:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329152 from it ffffb5d404243dc8 00010000:00010000:2.0:1787695652.991288:0:98951:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695652.991296:0:98951:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695652.991304:0:98951:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695652.991310:0:98951:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695652.991339:0:98951:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.991383:0:98951:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695652.991391:0:98951:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695652.991394:0:98951:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329152 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee0e expref: -99 pid: 98951 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695652.991438:0:98951:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695652.991446:0:98951:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f57480 x1874531134067840/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695652.991470:0:98951:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f57480 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98951:1874531134067840:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695652.991560:0:98951:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695652.997104:0:98951:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f57480 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98951:1874531134067840:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695652.997161:0:98951:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54d:0x0](ffff9d85b3cf2a08) 00000080:00200000:2.0:1787695652.997168:0:98951:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54d:0x0], updating i_size 512 00000080:00200000:2.0:1787695653.005569:0:98952:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.005600:0:98952:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.005614:0:98952:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.005784:0:98952:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98952 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.005795:0:98952:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98952 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.005805:0:98952:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98952 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.005830:0:98952:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98952:1874531134067968:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.006246:0:98952:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.010684:0:98952:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98952:1874531134067968:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.010716:0:98952:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.010736:0:98952:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.010750:0:98952:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.010762:0:98952:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.010776:0:98952:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.010783:0:98952:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.010791:0:98952:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.010803:0:98952:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55c00 x1874531134067968/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.010816:0:98952:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.010830:0:98952:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.010837:0:98952:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.011210:0:98952:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.011219:0:98952:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329159 00000080:00010000:2.0:1787695653.011226:0:98952:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.011230:0:98952:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329159 from it ffffb5d40424bc20 00010000:00010000:2.0:1787695653.011233:0:98952:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.011244:0:98952:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.011271:0:98952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.011298:0:98952:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.011302:0:98952:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.011323:0:98952:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.011364:0:98952:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98952 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.011372:0:98952:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98952 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.011382:0:98952:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98952 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.011402:0:98952:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98952:1874531134068096:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.011492:0:98952:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.014580:0:98952:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98952:1874531134068096:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.014611:0:98952:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014651:0:98952:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014660:0:98952:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014671:0:98952:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014682:0:98952:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014695:0:98952:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014704:0:98952:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014713:0:98952:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.014725:0:98952:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55c00 x1874531134068096/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.014738:0:98952:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014755:0:98952:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1357 (0 0) 00000080:00200000:2.0:1787695653.014762:0:98952:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695653.014791:0:98952:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54d:0x0](ffff9d85b3cf63c8) 00000080:00200000:2.0:1787695653.014799:0:98952:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54d:0x0], updating i_size 512 00000080:00010000:2.0:1787695653.014806:0:98952:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x54d:0x0](ffff9d85b3cf63c8) for lock 0x678ae32e95329160 00000080:00010000:2.0:1787695653.014817:0:98952:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x54d:0x0](ffff9d85b3cf63c8) 00000080:00010000:2.0:1787695653.014821:0:98952:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329160 from it ffffb5d40424be08 00010000:00010000:2.0:1787695653.014824:0:98952:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.014833:0:98952:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.014897:0:98952:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.014913:0:98952:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.014923:0:98952:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.014935:0:98952:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.014942:0:98952:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.015194:0:98952:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.015198:0:98952:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329159 00000080:00010000:2.0:1787695653.015204:0:98952:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329159 from it ffffb5d40424bd90 00010000:00010000:2.0:1787695653.015207:0:98952:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.015215:0:98952:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.015224:0:98952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.015233:0:98952:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695653.015256:0:98952:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.015291:0:98952:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.015327:0:98952:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.015335:0:98952:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.015341:0:98952:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695653.015344:0:98952:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329160 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x54d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee46 expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.015360:0:98952:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329159 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee2a expref: -99 pid: 98952 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695653.015376:0:98952:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.015384:0:98952:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f55c00 x1874531134068352/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.015413:0:98952:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98952:1874531134068352:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695653.015618:0:98952:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.023317:0:98952:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98952:1874531134068352:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695653.023588:0:98952:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x54d:0x0](ffff9d85b3cf63c8) 00000080:00200000:2.0:1787695653.062290:0:98954:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.062322:0:98954:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.062335:0:98954:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.062387:0:98954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98954 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.062396:0:98954:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98954 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.062406:0:98954:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98954 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.062444:0:98954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98954:1874531134068864:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.062591:0:98954:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.066294:0:98954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98954:1874531134068864:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.066323:0:98954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.066335:0:98954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.066350:0:98954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.066362:0:98954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.066374:0:98954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.066382:0:98954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.066389:0:98954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.066400:0:98954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55880 x1874531134068864/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.066414:0:98954:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.066440:0:98954:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.066447:0:98954:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695653.066568:0:98954:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695653.066578:0:98954:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532916e 00000080:00010000:0.0:1787695653.066585:0:98954:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.066588:0:98954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532916e from it ffffb5d404263c20 00010000:00010000:0.0:1787695653.066593:0:98954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.066603:0:98954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.066699:0:98954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695653.066724:0:98954:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695653.066728:0:98954:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695653.066749:0:98954:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695653.066795:0:98954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98954 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.066803:0:98954:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98954 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.066813:0:98954:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98954 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.066834:0:98954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98954:1874531134068992:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.067513:0:98954:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.071620:0:98954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98954:1874531134068992:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695653.071737:0:98954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.071761:0:98954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.071771:0:98954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.071783:0:98954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.071795:0:98954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.071809:0:98954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.071818:0:98954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.071826:0:98954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.071838:0:98954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3b480 x1874531134068992/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.072046:0:98954:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.072060:0:98954:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1358 (0 0) 00000080:00200000:0.0:1787695653.072069:0:98954:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695653.072182:0:98954:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54e:0x0](ffff9d85b3cd00c8) 00000080:00200000:0.0:1787695653.072192:0:98954:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54e:0x0], updating i_size 512 00000080:00010000:0.0:1787695653.072199:0:98954:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x54e:0x0](ffff9d85b3cd00c8) for lock 0x678ae32e95329175 00000080:00010000:0.0:1787695653.072212:0:98954:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x54e:0x0](ffff9d85b3cd00c8) 00000080:00010000:0.0:1787695653.072216:0:98954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329175 from it ffffb5d404263e08 00010000:00010000:0.0:1787695653.072220:0:98954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.072230:0:98954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.072264:0:98954:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.072281:0:98954:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.072293:0:98954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.072303:0:98954:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.072309:0:98954:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.072314:0:98954:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.072316:0:98954:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532916e 00000080:00010000:0.0:1787695653.072320:0:98954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532916e from it ffffb5d404263d90 00010000:00010000:0.0:1787695653.072323:0:98954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.072330:0:98954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.072338:0:98954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.072345:0:98954:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695653.072365:0:98954:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.072402:0:98954:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.072450:0:98954:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.072458:0:98954:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.072464:0:98954:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695653.072467:0:98954:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329175 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x54e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeea1 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.072485:0:98954:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e9532916e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eee85 expref: -99 pid: 98954 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695653.072500:0:98954:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695653.072508:0:98954:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3b480 x1874531134069248/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695653.072531:0:98954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3b480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98954:1874531134069248:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695653.072666:0:98954:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00010000:00010000:0.0:1787695653.077102:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787695653.077939:0:98954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98954:1874531134069248:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695653.077989:0:98954:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x54e:0x0](ffff9d85b3cd00c8) 00000080:00200000:0.0:1787695653.088522:0:98955:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.088553:0:98955:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.088677:0:98955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98955 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.088691:0:98955:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98955 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.088704:0:98955:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98955 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.088737:0:98955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98955:1874531134069376:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695653.089235:0:98955:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.093342:0:98955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98955:1874531134069376:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695653.093373:0:98955:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093387:0:98955:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093399:0:98955:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093410:0:98955:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093436:0:98955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093445:0:98955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093459:0:98955:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.093474:0:98955:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f38000 x1874531134069376/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.093491:0:98955:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093507:0:98955:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695653.093514:0:98955:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695653.093542:0:98955:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695653.093548:0:98955:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532917c 00000080:00010000:0.0:1787695653.093553:0:98955:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.093556:0:98955:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532917c from it ffffb5d4041e3be0 00010000:00010000:0.0:1787695653.093559:0:98955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.093567:0:98955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.093589:0:98955:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695653.093608:0:98955:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695653.093612:0:98955:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.093624:0:98955:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.093632:0:98955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.093647:0:98955:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.093653:0:98955:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.093656:0:98955:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.093659:0:98955:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532917c 00000080:00010000:0.0:1787695653.093662:0:98955:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532917c from it ffffb5d4041e3ce0 00010000:00010000:0.0:1787695653.093665:0:98955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.093672:0:98955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.093679:0:98955:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.093686:0:98955:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.093691:0:98955:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.093696:0:98955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.093704:0:98955:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.093710:0:98955:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.093713:0:98955:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.093716:0:98955:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532917c 00000080:00010000:0.0:1787695653.093719:0:98955:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532917c from it ffffb5d4041e3dc8 00010000:00010000:0.0:1787695653.093723:0:98955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.093730:0:98955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695653.093737:0:98955:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.093742:0:98955:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695653.093773:0:98955:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695653.093816:0:98955:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695653.093822:0:98955:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695653.093825:0:98955:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e9532917c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeec4 expref: -99 pid: 98955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695653.094341:0:98955:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695653.094353:0:98955:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f38000 x1874531134069632/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695653.094379:0:98955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98955:1874531134069632:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695653.094476:0:98955:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.100511:0:98955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f38000 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98955:1874531134069632:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695653.100565:0:98955:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54f:0x0](ffff9d85b3cd00c8) 00000080:00200000:0.0:1787695653.100575:0:98955:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54f:0x0], updating i_size 512 00000080:00200000:0.0:1787695653.110213:0:98956:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.110239:0:98956:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.110254:0:98956:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695653.110311:0:98956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98956 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.110321:0:98956:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98956 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.110331:0:98956:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98956 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.110356:0:98956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98956:1874531134069760:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.110501:0:98956:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.117942:0:98956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98956:1874531134069760:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695653.117980:0:98956:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.117992:0:98956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118006:0:98956:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118017:0:98956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118030:0:98956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118044:0:98956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118052:0:98956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.118064:0:98956:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39f80 x1874531134069760/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.118077:0:98956:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118099:0:98956:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695653.118106:0:98956:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695653.118132:0:98956:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695653.118138:0:98956:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329183 00000080:00010000:0.0:1787695653.118144:0:98956:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.118147:0:98956:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329183 from it ffffb5d404233c20 00010000:00010000:0.0:1787695653.118151:0:98956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118160:0:98956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.118186:0:98956:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695653.118206:0:98956:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695653.118211:0:98956:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695653.118229:0:98956:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695653.118268:0:98956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98956 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118275:0:98956:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98956 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.118284:0:98956:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98956 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.118303:0:98956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98956:1874531134069888:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.118365:0:98956:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.125490:0:98956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98956:1874531134069888:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695653.125518:0:98956:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125539:0:98956:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125548:0:98956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 4/1,0 mode: --/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125560:0:98956:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125570:0:98956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125583:0:98956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125592:0:98956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125601:0:98956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.125613:0:98956:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39f80 x1874531134069888/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.125626:0:98956:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125636:0:98956:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1359 (0 0) 00000080:00200000:0.0:1787695653.125643:0:98956:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695653.125679:0:98956:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x54f:0x0](ffff9d85b3cd0908) 00000080:00200000:0.0:1787695653.125687:0:98956:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x54f:0x0], updating i_size 512 00000080:00010000:0.0:1787695653.125694:0:98956:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x54f:0x0](ffff9d85b3cd0908) for lock 0x678ae32e9532918a 00000080:00010000:0.0:1787695653.125706:0:98956:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x54f:0x0](ffff9d85b3cd0908) 00000080:00010000:0.0:1787695653.125710:0:98956:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532918a from it ffffb5d404233e08 00010000:00010000:0.0:1787695653.125714:0:98956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.125724:0:98956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.125759:0:98956:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.125775:0:98956:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.125786:0:98956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.125797:0:98956:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.125804:0:98956:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.125809:0:98956:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.125812:0:98956:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329183 00000080:00010000:0.0:1787695653.125816:0:98956:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329183 from it ffffb5d404233d90 00010000:00010000:0.0:1787695653.125819:0:98956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.125826:0:98956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.125835:0:98956:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.125842:0:98956:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695653.126259:0:98956:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.126296:0:98956:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.126333:0:98956:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 1/0,0 mode: --/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.126342:0:98956:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.126353:0:98956:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695653.126356:0:98956:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898da00/0x678ae32e9532918a lrc: 0/0,0 mode: --/PR res: [0x200000401:0x54f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeefc expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.126373:0:98956:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e95329183 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeee0 expref: -99 pid: 98956 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695653.126390:0:98956:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695653.126397:0:98956:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f39f80 x1874531134070144/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695653.126431:0:98956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98956:1874531134070144:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695653.126563:0:98956:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.132566:0:98956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39f80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98956:1874531134070144:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695653.145588:0:98957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.145616:0:98957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.145682:0:98957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.145693:0:98957:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.145709:0:98957:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98957 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.145734:0:98957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98957:1874531134070272:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.145905:0:98957:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.149703:0:98957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98957:1874531134070272:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.149733:0:98957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.149744:0:98957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.149756:0:98957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.149765:0:98957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.149776:0:98957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.149783:0:98957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.149789:0:98957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.149806:0:98957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55880 x1874531134070272/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.149819:0:98957:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329191 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef1f expref: -99 pid: 98957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.149831:0:98957:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.149839:0:98957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.166907:0:98958:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.166948:0:98958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.166966:0:98958:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.167034:0:98958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98958 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.167045:0:98958:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98958 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.167056:0:98958:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98958 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.167085:0:98958:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98958:1874531134070656:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.167181:0:98958:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.172307:0:98958:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98958:1874531134070656:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.172343:0:98958:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172357:0:98958:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172372:0:98958:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172385:0:98958:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172399:0:98958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172409:0:98958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172429:0:98958:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.172442:0:98958:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55880 x1874531134070656/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.172457:0:98958:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172473:0:98958:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.172482:0:98958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.172513:0:98958:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.172520:0:98958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329198 00000080:00010000:2.0:1787695653.172525:0:98958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.172528:0:98958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329198 from it ffffb5d40420bc20 00010000:00010000:2.0:1787695653.172533:0:98958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172542:0:98958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.172572:0:98958:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.172596:0:98958:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.172600:0:98958:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.172621:0:98958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.172666:0:98958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98958 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172673:0:98958:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98958 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.172682:0:98958:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98958 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.172701:0:98958:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98958:1874531134070784:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.172783:0:98958:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.177324:0:98958:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98958:1874531134070784:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.177357:0:98958:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177382:0:98958:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 4/1,0 mode: --/PR res: [0x200000401:0x550:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177392:0:98958:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 4/1,0 mode: --/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177406:0:98958:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177428:0:98958:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177441:0:98958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177450:0:98958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177457:0:98958:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.177471:0:98958:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0f55880 x1874531134070784/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.177484:0:98958:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177496:0:98958:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1360 (0 0) 00000080:00200000:2.0:1787695653.177503:0:98958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695653.177543:0:98958:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x550:0x0](ffff9d85b3cf63c8) 00000080:00200000:2.0:1787695653.177553:0:98958:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x550:0x0], updating i_size 512 00000080:00010000:2.0:1787695653.177561:0:98958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x550:0x0](ffff9d85b3cf63c8) for lock 0x678ae32e9532919f 00000080:00010000:2.0:1787695653.177573:0:98958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x550:0x0](ffff9d85b3cf63c8) 00000080:00010000:2.0:1787695653.177577:0:98958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532919f from it ffffb5d40420be08 00010000:00010000:2.0:1787695653.177581:0:98958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.177590:0:98958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.177625:0:98958:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.177642:0:98958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.177652:0:98958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.177662:0:98958:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.177669:0:98958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.177672:0:98958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.177676:0:98958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329198 00000080:00010000:2.0:1787695653.177680:0:98958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329198 from it ffffb5d40420bd90 00010000:00010000:2.0:1787695653.177683:0:98958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.177691:0:98958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.177699:0:98958:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.177706:0:98958:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695653.177727:0:98958:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.177761:0:98958:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.177792:0:98958:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 1/0,0 mode: --/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.177801:0:98958:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.177808:0:98958:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695653.177812:0:98958:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532919f lrc: 0/0,0 mode: --/PR res: [0x200000401:0x550:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eef57 expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.177827:0:98958:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329198 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eef3b expref: -99 pid: 98958 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695653.177843:0:98958:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.177890:0:98958:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0f55880 x1874531134071040/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.177916:0:98958:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98958:1874531134071040:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695653.178021:0:98958:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.184068:0:98958:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0f55880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98958:1874531134071040:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695653.184126:0:98958:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x550:0x0](ffff9d85b3cf63c8) 00000080:00200000:0.0:1787695653.198846:0:98959:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.199316:0:98959:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.199396:0:98959:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e953291a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98959 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.199406:0:98959:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e953291a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98959 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.199428:0:98959:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898ca00/0x678ae32e953291a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98959 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.199456:0:98959:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98959:1874531134071168:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695653.199605:0:98959:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1d80 going to sleep for 12 seconds 00000080:00200000:0.0:1787695653.246513:0:98960:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x551:0x0](ffff9d85b3ce2a08) 00000080:00200000:0.0:1787695653.246525:0:98960:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x551:0x0], updating i_size 512 00000080:00010000:0.0:1787695653.246535:0:98960:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x551:0x0](ffff9d85b3ce2a08) for lock 0x678ae32e953291b4 00000080:00010000:0.0:1787695653.246546:0:98960:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x551:0x0](ffff9d85b3ce2a08) 00000080:00010000:0.0:1787695653.246550:0:98960:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291b4 from it ffffb5d404273e08 00010000:00010000:0.0:1787695653.246554:0:98960:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e953291b4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x551:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefb2 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.246565:0:98960:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e953291b4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x551:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefb2 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.246599:0:98960:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.246617:0:98960:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.246641:0:98960:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e953291ad lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef96 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.246655:0:98960:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e953291ad lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef96 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.246662:0:98960:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.246668:0:98960:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.246671:0:98960:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953291ad 00000080:00010000:0.0:1787695653.246675:0:98960:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291ad from it ffffb5d404273d90 00010000:00010000:0.0:1787695653.246679:0:98960:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e953291ad lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eef96 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.246687:0:98960:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e953291ad lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eef96 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.246702:0:98960:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.246710:0:98960:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695653.246728:0:98960:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e953291ad lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eef96 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.246768:0:98960:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e953291b4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x551:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eefb2 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.246807:0:98960:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e953291b4 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x551:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eefb2 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.246816:0:98960:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e953291ad lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eef96 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.246822:0:98960:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695653.246825:0:98960:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030a00/0x678ae32e953291b4 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x551:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eefb2 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.246843:0:98960:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e953291ad lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eef96 expref: -99 pid: 98960 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695653.246905:0:98960:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695653.246916:0:98960:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f38a80 x1874531134071936/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695653.247214:0:98960:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38a80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98960:1874531134071936:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695653.247314:0:98960:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1780 going to sleep for 12 seconds 00000080:00200000:2.0:1787695653.253362:0:98960:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x551:0x0](ffff9d85b3ce2a08) 00000080:00200000:2.0:1787695653.264546:0:98961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.264579:0:98961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.264680:0:98961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.264691:0:98961:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.264702:0:98961:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98961 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.264730:0:98961:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98961:1874531134072064:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.264838:0:98961:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.268614:0:98961:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98961:1874531134072064:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.268760:0:98961:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.268774:0:98961:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.268786:0:98961:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.268798:0:98961:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.268812:0:98961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.268820:0:98961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.268835:0:98961:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.268847:0:98961:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb0e00 x1874531134072064/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.269230:0:98961:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.269242:0:98961:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.269249:0:98961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.269276:0:98961:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.269283:0:98961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291bb 00000080:00010000:2.0:1787695653.269288:0:98961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.269291:0:98961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291bb from it ffffb5d40427bbe0 00010000:00010000:2.0:1787695653.269295:0:98961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.269304:0:98961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.269326:0:98961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.269346:0:98961:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.269350:0:98961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.269362:0:98961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.269371:0:98961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.269388:0:98961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.269395:0:98961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.269399:0:98961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.269402:0:98961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291bb 00000080:00010000:2.0:1787695653.269406:0:98961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291bb from it ffffb5d40427bce0 00010000:00010000:2.0:1787695653.269409:0:98961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.269430:0:98961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.269440:0:98961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.269447:0:98961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.269453:0:98961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.269459:0:98961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.269468:0:98961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.269475:0:98961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.269479:0:98961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.269482:0:98961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291bb 00000080:00010000:2.0:1787695653.269486:0:98961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291bb from it ffffb5d40427bdc8 00010000:00010000:2.0:1787695653.269490:0:98961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.269497:0:98961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.269506:0:98961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.269511:0:98961:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.269538:0:98961:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.269579:0:98961:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.269587:0:98961:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.269590:0:98961:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291bb lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eefd5 expref: -99 pid: 98961 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.269613:0:98961:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.269621:0:98961:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb0e00 x1874531134072320/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.269773:0:98961:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98961:1874531134072320:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.270070:0:98961:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.280107:0:98961:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98961:1874531134072320:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.280159:0:98961:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x552:0x0](ffff9d85b3cf63c8) 00000080:00200000:2.0:1787695653.280166:0:98961:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x552:0x0], updating i_size 512 00000080:00200000:0.0:1787695653.288838:0:98962:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.289280:0:98962:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.289297:0:98962:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695653.289357:0:98962:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98962 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.289367:0:98962:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98962 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.289378:0:98962:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98962 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.289402:0:98962:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3bb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98962:1874531134072448:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.289556:0:98962:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c19c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.292464:0:98962:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3bb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98962:1874531134072448:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695653.292490:0:98962:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292503:0:98962:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292515:0:98962:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292526:0:98962:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292539:0:98962:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292553:0:98962:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292562:0:98962:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.292574:0:98962:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3bb80 x1874531134072448/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.292587:0:98962:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292600:0:98962:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695653.292607:0:98962:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695653.292690:0:98962:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695653.292697:0:98962:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953291c2 00000080:00010000:0.0:1787695653.292703:0:98962:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.292707:0:98962:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291c2 from it ffffb5d404233c20 00010000:00010000:0.0:1787695653.292711:0:98962:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292721:0:98962:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.292745:0:98962:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695653.292767:0:98962:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695653.292772:0:98962:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695653.292791:0:98962:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695653.292828:0:98962:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98962 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292836:0:98962:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98962 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.292846:0:98962:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98962 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.293287:0:98962:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3bb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98962:1874531134072576:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.293392:0:98962:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c19c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.296595:0:98962:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3bb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98962:1874531134072576:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695653.296621:0:98962:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296740:0:98962:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x552:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296750:0:98962:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296763:0:98962:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296775:0:98962:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296789:0:98962:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296797:0:98962:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296804:0:98962:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.296817:0:98962:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f3bb80 x1874531134072576/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.296830:0:98962:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.296839:0:98962:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1362 (0 0) 00000080:00200000:0.0:1787695653.296845:0:98962:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695653.297409:0:98962:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x552:0x0](ffff9d85b3cd1148) 00000080:00200000:0.0:1787695653.297431:0:98962:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x552:0x0], updating i_size 512 00000080:00010000:0.0:1787695653.297439:0:98962:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x552:0x0](ffff9d85b3cd1148) for lock 0x678ae32e953291c9 00000080:00010000:0.0:1787695653.297451:0:98962:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x552:0x0](ffff9d85b3cd1148) 00000080:00010000:0.0:1787695653.297455:0:98962:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291c9 from it ffffb5d404233e08 00010000:00010000:0.0:1787695653.297459:0:98962:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.297469:0:98962:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.297499:0:98962:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.297516:0:98962:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.297528:0:98962:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.297540:0:98962:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.297547:0:98962:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.297551:0:98962:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.297555:0:98962:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953291c2 00000080:00010000:0.0:1787695653.297558:0:98962:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291c2 from it ffffb5d404233d90 00010000:00010000:0.0:1787695653.297562:0:98962:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.297569:0:98962:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.297578:0:98962:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.297585:0:98962:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695653.297604:0:98962:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.297649:0:98962:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.297685:0:98962:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.297694:0:98962:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.297706:0:98962:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695653.297709:0:98962:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d200/0x678ae32e953291c9 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x552:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef00d expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.297723:0:98962:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d400/0x678ae32e953291c2 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832eeff1 expref: -99 pid: 98962 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695653.297737:0:98962:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695653.297745:0:98962:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3bb80 x1874531134072832/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695653.297766:0:98962:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3bb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98962:1874531134072832:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695653.297826:0:98962:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c19c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.303040:0:98962:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3bb80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98962:1874531134072832:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695653.303092:0:98962:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x552:0x0](ffff9d85b3cd1148) 00000080:00200000:0.0:1787695653.311784:0:98963:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.311808:0:98963:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.316343:0:98963:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98963 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.316360:0:98963:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98963 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.316370:0:98963:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98963 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.316394:0:98963:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98963:1874531134072960:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695653.316500:0:98963:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c19c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.320664:0:98963:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98963:1874531134072960:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695653.320694:0:98963:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.320706:0:98963:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.320719:0:98963:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.320730:0:98963:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.320742:0:98963:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.320751:0:98963:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.320759:0:98963:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.320771:0:98963:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39180 x1874531134072960/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.320786:0:98963:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.320804:0:98963:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695653.320812:0:98963:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695653.320842:0:98963:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695653.321183:0:98963:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291d0 00000080:00010000:0.0:1787695653.321192:0:98963:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.321195:0:98963:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291d0 from it ffffb5d40426bbe0 00010000:00010000:0.0:1787695653.321199:0:98963:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.321209:0:98963:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.321236:0:98963:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695653.321259:0:98963:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695653.321265:0:98963:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.321279:0:98963:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.321288:0:98963:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.321299:0:98963:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.321306:0:98963:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.321311:0:98963:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.321314:0:98963:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291d0 00000080:00010000:0.0:1787695653.321317:0:98963:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291d0 from it ffffb5d40426bce0 00010000:00010000:0.0:1787695653.321321:0:98963:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.321329:0:98963:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.321338:0:98963:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.321344:0:98963:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.321350:0:98963:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.321355:0:98963:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.321364:0:98963:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.321371:0:98963:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.321374:0:98963:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.321377:0:98963:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291d0 00000080:00010000:0.0:1787695653.321381:0:98963:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291d0 from it ffffb5d40426bdc8 00010000:00010000:0.0:1787695653.321384:0:98963:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.321392:0:98963:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695653.321400:0:98963:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.321405:0:98963:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695653.321445:0:98963:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695653.321490:0:98963:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695653.321497:0:98963:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695653.321501:0:98963:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d400/0x678ae32e953291d0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef030 expref: -99 pid: 98963 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695653.321527:0:98963:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695653.321536:0:98963:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f39180 x1874531134073216/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695653.321558:0:98963:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98963:1874531134073216:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695653.321757:0:98963:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c19c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.327463:0:98963:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98963:1874531134073216:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695653.327513:0:98963:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x553:0x0](ffff9d85b3cd1148) 00000080:00200000:0.0:1787695653.327520:0:98963:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x553:0x0], updating i_size 512 00000080:00200000:2.0:1787695653.334615:0:98964:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.335582:0:98964:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.335598:0:98964:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.335665:0:98964:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98964 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.335676:0:98964:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98964 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.335686:0:98964:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98964 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.335712:0:98964:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98964:1874531134073344:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.335793:0:98964:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.339496:0:98964:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98964:1874531134073344:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.339521:0:98964:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339534:0:98964:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339546:0:98964:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339557:0:98964:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339570:0:98964:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339585:0:98964:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339593:0:98964:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.339604:0:98964:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb3480 x1874531134073344/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.339618:0:98964:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339638:0:98964:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.339645:0:98964:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.339673:0:98964:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.339679:0:98964:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953291d7 00000080:00010000:2.0:1787695653.339684:0:98964:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.339688:0:98964:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291d7 from it ffffb5d404283c20 00010000:00010000:2.0:1787695653.339692:0:98964:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339701:0:98964:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.339726:0:98964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.339747:0:98964:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.339752:0:98964:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.339769:0:98964:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.339805:0:98964:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98964 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339813:0:98964:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98964 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.339822:0:98964:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98964 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.339841:0:98964:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98964:1874531134073472:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.340164:0:98964:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.343751:0:98964:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98964:1874531134073472:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.343776:0:98964:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343798:0:98964:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 4/1,0 mode: --/PR res: [0x200000401:0x553:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343807:0:98964:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 4/1,0 mode: --/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343819:0:98964:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343832:0:98964:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343845:0:98964:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343895:0:98964:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343903:0:98964:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.343916:0:98964:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb3480 x1874531134073472/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.343929:0:98964:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.343939:0:98964:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1363 (0 0) 00000080:00200000:2.0:1787695653.343955:0:98964:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695653.344014:0:98964:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x553:0x0](ffff9d85b3cfa1c8) 00000080:00200000:2.0:1787695653.344023:0:98964:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x553:0x0], updating i_size 512 00000080:00010000:2.0:1787695653.344031:0:98964:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x553:0x0](ffff9d85b3cfa1c8) for lock 0x678ae32e953291de 00000080:00010000:2.0:1787695653.344041:0:98964:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x553:0x0](ffff9d85b3cfa1c8) 00000080:00010000:2.0:1787695653.344045:0:98964:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291de from it ffffb5d404283e08 00010000:00010000:2.0:1787695653.344049:0:98964:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.344059:0:98964:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.344351:0:98964:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.344369:0:98964:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.344380:0:98964:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.344391:0:98964:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.344398:0:98964:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.344403:0:98964:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.344406:0:98964:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953291d7 00000080:00010000:2.0:1787695653.344420:0:98964:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291d7 from it ffffb5d404283d90 00010000:00010000:2.0:1787695653.344424:0:98964:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.344431:0:98964:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.344440:0:98964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.344448:0:98964:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695653.344469:0:98964:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.344501:0:98964:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.344538:0:98964:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 1/0,0 mode: --/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.344545:0:98964:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.344556:0:98964:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695653.344559:0:98964:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291de lrc: 0/0,0 mode: --/PR res: [0x200000401:0x553:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef068 expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.344575:0:98964:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291d7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef04c expref: -99 pid: 98964 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695653.344591:0:98964:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.344600:0:98964:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb3480 x1874531134073728/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.344624:0:98964:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98964:1874531134073728:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695653.344696:0:98964:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.349883:0:98964:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98964:1874531134073728:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695653.349928:0:98964:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x553:0x0](ffff9d85b3cfa1c8) 00000080:00200000:2.0:1787695653.358288:0:98965:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.358319:0:98965:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.358384:0:98965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98965 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.358396:0:98965:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98965 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.358407:0:98965:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98965 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.358450:0:98965:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98965:1874531134073856:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.358602:0:98965:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.362149:0:98965:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98965:1874531134073856:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.362178:0:98965:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362192:0:98965:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362204:0:98965:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362215:0:98965:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362228:0:98965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362237:0:98965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362244:0:98965:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.362255:0:98965:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb1f80 x1874531134073856/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.362269:0:98965:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362288:0:98965:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.362296:0:98965:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.362324:0:98965:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.362331:0:98965:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291e5 00000080:00010000:2.0:1787695653.362337:0:98965:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.362339:0:98965:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291e5 from it ffffb5d40428bbe0 00010000:00010000:2.0:1787695653.362342:0:98965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.362351:0:98965:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.362374:0:98965:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.362397:0:98965:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.362402:0:98965:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.362428:0:98965:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.362439:0:98965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.362450:0:98965:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.362458:0:98965:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.362462:0:98965:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.362466:0:98965:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291e5 00000080:00010000:2.0:1787695653.362470:0:98965:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291e5 from it ffffb5d40428bce0 00010000:00010000:2.0:1787695653.362474:0:98965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.362482:0:98965:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.362491:0:98965:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.362498:0:98965:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.362504:0:98965:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.362510:0:98965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.362519:0:98965:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.362526:0:98965:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.362530:0:98965:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.362533:0:98965:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291e5 00000080:00010000:2.0:1787695653.362536:0:98965:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291e5 from it ffffb5d40428bdc8 00010000:00010000:2.0:1787695653.362540:0:98965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.362547:0:98965:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.362555:0:98965:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.362561:0:98965:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.362591:0:98965:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.362781:0:98965:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.362790:0:98965:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.362793:0:98965:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953291e5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef08b expref: -99 pid: 98965 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.362822:0:98965:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.362830:0:98965:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb1f80 x1874531134074112/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.363043:0:98965:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98965:1874531134074112:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.363244:0:98965:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000080:00200000:2.0:1787695653.380692:0:98966:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.380721:0:98966:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.380735:0:98966:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.380787:0:98966:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98966 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.380797:0:98966:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98966 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.380807:0:98966:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98966 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.380831:0:98966:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98966:1874531134074240:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.381280:0:98966:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.384682:0:98966:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98966:1874531134074240:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.384709:0:98966:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.384722:0:98966:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.384734:0:98966:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.384745:0:98966:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.384758:0:98966:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.384767:0:98966:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.384774:0:98966:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.384786:0:98966:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb0000 x1874531134074240/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.384800:0:98966:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.384813:0:98966:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.384820:0:98966:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.384846:0:98966:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.385340:0:98966:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953291ec 00000080:00010000:2.0:1787695653.385346:0:98966:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.385350:0:98966:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291ec from it ffffb5d404233c20 00010000:00010000:2.0:1787695653.385354:0:98966:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.385363:0:98966:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.385389:0:98966:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.385424:0:98966:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.385429:0:98966:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.385454:0:98966:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.385497:0:98966:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98966 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.385504:0:98966:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98966 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.385513:0:98966:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98966 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.385536:0:98966:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98966:1874531134074368:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.385621:0:98966:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.390562:0:98966:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98966:1874531134074368:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.390591:0:98966:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390617:0:98966:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x554:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390625:0:98966:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390636:0:98966:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390648:0:98966:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390661:0:98966:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390670:0:98966:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390677:0:98966:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.390689:0:98966:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb0000 x1874531134074368/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.390701:0:98966:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390717:0:98966:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1364 (0 0) 00000080:00200000:2.0:1787695653.390724:0:98966:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695653.390756:0:98966:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x554:0x0](ffff9d85b3cfa1c8) 00000080:00200000:2.0:1787695653.390764:0:98966:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x554:0x0], updating i_size 512 00000080:00010000:2.0:1787695653.390771:0:98966:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x554:0x0](ffff9d85b3cfa1c8) for lock 0x678ae32e953291f3 00000080:00010000:2.0:1787695653.390781:0:98966:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x554:0x0](ffff9d85b3cfa1c8) 00000080:00010000:2.0:1787695653.390785:0:98966:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291f3 from it ffffb5d404233e08 00010000:00010000:2.0:1787695653.390789:0:98966:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.390798:0:98966:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.390828:0:98966:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.390844:0:98966:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.390886:0:98966:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.390899:0:98966:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.390906:0:98966:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.390911:0:98966:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.390915:0:98966:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953291ec 00000080:00010000:2.0:1787695653.390919:0:98966:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291ec from it ffffb5d404233d90 00010000:00010000:2.0:1787695653.390922:0:98966:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.390931:0:98966:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.391239:0:98966:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.391248:0:98966:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695653.391267:0:98966:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.391298:0:98966:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.391335:0:98966:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.391344:0:98966:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.391351:0:98966:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695653.391355:0:98966:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953291f3 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x554:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef0c3 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.391370:0:98966:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953291ec lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef0a7 expref: -99 pid: 98966 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695653.391387:0:98966:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.391394:0:98966:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb0000 x1874531134074624/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.391435:0:98966:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98966:1874531134074624:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695653.391508:0:98966:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.397609:0:98966:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98966:1874531134074624:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695653.397665:0:98966:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x554:0x0](ffff9d85b3cfa1c8) 00000080:00200000:2.0:1787695653.408386:0:98967:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.408423:0:98967:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.408487:0:98967:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98967 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.408498:0:98967:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98967 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.408509:0:98967:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98967 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.408533:0:98967:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98967:1874531134074752:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.408677:0:98967:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.412403:0:98967:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98967:1874531134074752:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.412441:0:98967:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412455:0:98967:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412468:0:98967:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412486:0:98967:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412499:0:98967:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412508:0:98967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412516:0:98967:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.412529:0:98967:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb0380 x1874531134074752/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.412542:0:98967:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412554:0:98967:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.412561:0:98967:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.412589:0:98967:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.412597:0:98967:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291fa 00000080:00010000:2.0:1787695653.412603:0:98967:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.412606:0:98967:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291fa from it ffffb5d40428bbe0 00010000:00010000:2.0:1787695653.412610:0:98967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.412703:0:98967:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.412729:0:98967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.412752:0:98967:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.412757:0:98967:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.412769:0:98967:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.412778:0:98967:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.412789:0:98967:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.412796:0:98967:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.412801:0:98967:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.412804:0:98967:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291fa 00000080:00010000:2.0:1787695653.412808:0:98967:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291fa from it ffffb5d40428bce0 00010000:00010000:2.0:1787695653.412812:0:98967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.412820:0:98967:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.412828:0:98967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.412835:0:98967:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.412841:0:98967:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.412847:0:98967:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.413143:0:98967:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.413152:0:98967:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.413157:0:98967:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.413160:0:98967:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953291fa 00000080:00010000:2.0:1787695653.413164:0:98967:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953291fa from it ffffb5d40428bdc8 00010000:00010000:2.0:1787695653.413168:0:98967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.413176:0:98967:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.413186:0:98967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.413191:0:98967:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.413222:0:98967:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.413264:0:98967:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.413271:0:98967:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.413274:0:98967:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953291fa lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef0e6 expref: -99 pid: 98967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.413296:0:98967:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.413305:0:98967:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb0380 x1874531134075008/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.413325:0:98967:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98967:1874531134075008:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.413458:0:98967:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.424063:0:98967:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98967:1874531134075008:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.424172:0:98967:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x555:0x0](ffff9d85b3cfa1c8) 00000080:00200000:2.0:1787695653.424180:0:98967:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x555:0x0], updating i_size 512 00000080:00200000:2.0:1787695653.431672:0:98968:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.431701:0:98968:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.431716:0:98968:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.431774:0:98968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98968 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.431785:0:98968:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98968 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.431795:0:98968:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98968 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.431823:0:98968:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98968:1874531134075136:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.432278:0:98968:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.436364:0:98968:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98968:1874531134075136:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.436392:0:98968:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436405:0:98968:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436436:0:98968:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436448:0:98968:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436462:0:98968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436472:0:98968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436480:0:98968:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.436492:0:98968:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb1180 x1874531134075136/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.436506:0:98968:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436520:0:98968:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.436527:0:98968:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.436556:0:98968:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.436563:0:98968:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329201 00000080:00010000:2.0:1787695653.436568:0:98968:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.436571:0:98968:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329201 from it ffffb5d404293c20 00010000:00010000:2.0:1787695653.436575:0:98968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436584:0:98968:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.436609:0:98968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.436701:0:98968:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.436707:0:98968:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.436728:0:98968:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.436769:0:98968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98968 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436777:0:98968:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98968 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.436787:0:98968:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98968 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.436806:0:98968:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98968:1874531134075264:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.437229:0:98968:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.441143:0:98968:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98968:1874531134075264:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.441170:0:98968:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441192:0:98968:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x555:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441201:0:98968:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441213:0:98968:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441224:0:98968:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441237:0:98968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441245:0:98968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441253:0:98968:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.441265:0:98968:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb1180 x1874531134075264/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.441276:0:98968:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441286:0:98968:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1365 (0 0) 00000080:00200000:2.0:1787695653.441292:0:98968:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695653.441324:0:98968:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x555:0x0](ffff9d85b3cf80c8) 00000080:00200000:2.0:1787695653.441332:0:98968:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x555:0x0], updating i_size 512 00000080:00010000:2.0:1787695653.441339:0:98968:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x555:0x0](ffff9d85b3cf80c8) for lock 0x678ae32e95329208 00000080:00010000:2.0:1787695653.441350:0:98968:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x555:0x0](ffff9d85b3cf80c8) 00000080:00010000:2.0:1787695653.441354:0:98968:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329208 from it ffffb5d404293e08 00010000:00010000:2.0:1787695653.441357:0:98968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.441367:0:98968:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.441399:0:98968:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.441426:0:98968:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.441437:0:98968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.441449:0:98968:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.441456:0:98968:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.441461:0:98968:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.441464:0:98968:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329201 00000080:00010000:2.0:1787695653.441468:0:98968:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329201 from it ffffb5d404293d90 00010000:00010000:2.0:1787695653.441471:0:98968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.441479:0:98968:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.441487:0:98968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.441495:0:98968:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695653.441517:0:98968:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.441551:0:98968:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.441583:0:98968:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.441592:0:98968:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.441598:0:98968:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695653.441601:0:98968:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329208 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x555:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef11e expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.441617:0:98968:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329201 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef102 expref: -99 pid: 98968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695653.441696:0:98968:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.441704:0:98968:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb1180 x1874531134075520/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.441726:0:98968:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98968:1874531134075520:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695653.441881:0:98968:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.446768:0:98968:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1180 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98968:1874531134075520:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695653.446812:0:98968:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x555:0x0](ffff9d85b3cf80c8) 00000080:00200000:2.0:1787695653.456377:0:98969:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.456404:0:98969:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.456493:0:98969:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98969 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.456506:0:98969:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98969 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.456517:0:98969:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98969 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.456550:0:98969:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98969:1874531134075648:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.456814:0:98969:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.460255:0:98969:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98969:1874531134075648:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.460283:0:98969:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460297:0:98969:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460309:0:98969:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460320:0:98969:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460333:0:98969:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460342:0:98969:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460355:0:98969:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.460368:0:98969:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb0a80 x1874531134075648/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.460382:0:98969:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460394:0:98969:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.460401:0:98969:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.460439:0:98969:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.460446:0:98969:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532920f 00000080:00010000:2.0:1787695653.460452:0:98969:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.460455:0:98969:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532920f from it ffffb5d40429bbe0 00010000:00010000:2.0:1787695653.460459:0:98969:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.460468:0:98969:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.460492:0:98969:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.460514:0:98969:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.460519:0:98969:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.460531:0:98969:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.460539:0:98969:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.460557:0:98969:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.460564:0:98969:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.460568:0:98969:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.460571:0:98969:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532920f 00000080:00010000:2.0:1787695653.460575:0:98969:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532920f from it ffffb5d40429bce0 00010000:00010000:2.0:1787695653.460579:0:98969:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.460586:0:98969:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.460595:0:98969:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.460601:0:98969:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.460607:0:98969:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.460612:0:98969:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.460621:0:98969:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.460627:0:98969:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.460631:0:98969:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.460634:0:98969:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532920f 00000080:00010000:2.0:1787695653.460637:0:98969:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532920f from it ffffb5d40429bdc8 00010000:00010000:2.0:1787695653.460641:0:98969:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.460648:0:98969:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.460656:0:98969:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.460662:0:98969:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.460689:0:98969:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.460731:0:98969:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.460738:0:98969:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.460741:0:98969:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e9532920f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef141 expref: -99 pid: 98969 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.460767:0:98969:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.460774:0:98969:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb0a80 x1874531134075904/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.460795:0:98969:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98969:1874531134075904:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.460912:0:98969:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000080:00200000:0.0:1787695653.472956:0:98970:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.472983:0:98970:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.472997:0:98970:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695653.473049:0:98970:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98970 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.473060:0:98970:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98970 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.473070:0:98970:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98970 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.473099:0:98970:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98970:1874531134076032:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.473182:0:98970:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c19c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.483592:0:98970:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98970:1874531134076160:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695653.483617:0:98970:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483692:0:98970:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x556:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483701:0:98970:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483713:0:98970:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483725:0:98970:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483738:0:98970:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483747:0:98970:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483755:0:98970:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.483773:0:98970:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2c380 x1874531134076160/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.483786:0:98970:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.483796:0:98970:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1366 (0 0) 00000080:00200000:0.0:1787695653.483803:0:98970:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695653.483835:0:98970:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x556:0x0](ffff9d85b3cd7448) 00000080:00200000:0.0:1787695653.483843:0:98970:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x556:0x0], updating i_size 512 00000080:00010000:0.0:1787695653.484281:0:98970:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x556:0x0](ffff9d85b3cd7448) for lock 0x678ae32e9532921d 00000080:00010000:0.0:1787695653.484298:0:98970:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x556:0x0](ffff9d85b3cd7448) 00000080:00010000:0.0:1787695653.484302:0:98970:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532921d from it ffffb5d4042a3e08 00010000:00010000:0.0:1787695653.484306:0:98970:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.484316:0:98970:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.484348:0:98970:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.484365:0:98970:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.484375:0:98970:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef15d expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.484387:0:98970:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef15d expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.484400:0:98970:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.484404:0:98970:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695653.484419:0:98970:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329216 00000080:00010000:0.0:1787695653.484424:0:98970:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329216 from it ffffb5d4042a3d90 00010000:00010000:0.0:1787695653.484427:0:98970:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef15d expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.484436:0:98970:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef15d expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.484445:0:98970:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.484453:0:98970:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695653.484474:0:98970:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef15d expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.484507:0:98970:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.484539:0:98970:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 1/0,0 mode: --/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.484548:0:98970:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef15d expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.484554:0:98970:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695653.484557:0:98970:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e9532921d lrc: 0/0,0 mode: --/PR res: [0x200000401:0x556:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef179 expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.484578:0:98970:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e95329216 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef15d expref: -99 pid: 98970 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695653.484593:0:98970:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695653.484601:0:98970:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3a680 x1874531134076416/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695653.484697:0:98970:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3a680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98970:1874531134076416:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695653.484811:0:98970:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1e40 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.491395:0:98970:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3a680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98970:1874531134076416:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695653.491453:0:98970:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x556:0x0](ffff9d85b3cd7448) 00000080:00200000:2.0:1787695653.503529:0:98971:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.503552:0:98971:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.503611:0:98971:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98971 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.503626:0:98971:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98971 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.503636:0:98971:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98971 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.503661:0:98971:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98971:1874531134076544:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.503790:0:98971:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.507737:0:98971:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98971:1874531134076544:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.507764:0:98971:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.507778:0:98971:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.507791:0:98971:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.507802:0:98971:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.507815:0:98971:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.507824:0:98971:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.507838:0:98971:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.508035:0:98971:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb1880 x1874531134076544/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.508051:0:98971:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.508065:0:98971:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.508072:0:98971:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.508224:0:98971:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.508230:0:98971:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329224 00000080:00010000:2.0:1787695653.508236:0:98971:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.508240:0:98971:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329224 from it ffffb5d4042abbe0 00010000:00010000:2.0:1787695653.508243:0:98971:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.508253:0:98971:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.508279:0:98971:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.508300:0:98971:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.508305:0:98971:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.508320:0:98971:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.508329:0:98971:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.508344:0:98971:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.508350:0:98971:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.508355:0:98971:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.508358:0:98971:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329224 00000080:00010000:2.0:1787695653.508361:0:98971:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329224 from it ffffb5d4042abce0 00010000:00010000:2.0:1787695653.508364:0:98971:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.508371:0:98971:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.508379:0:98971:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.508387:0:98971:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.508393:0:98971:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.508399:0:98971:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.508426:0:98971:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.508433:0:98971:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.508437:0:98971:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.508440:0:98971:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329224 00000080:00010000:2.0:1787695653.508443:0:98971:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329224 from it ffffb5d4042abdc8 00010000:00010000:2.0:1787695653.508448:0:98971:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.508455:0:98971:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.508463:0:98971:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.508469:0:98971:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.508503:0:98971:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.508547:0:98971:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.508555:0:98971:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.508558:0:98971:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329224 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef19c expref: -99 pid: 98971 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.508582:0:98971:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.508590:0:98971:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb1880 x1874531134076800/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.508614:0:98971:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98971:1874531134076800:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.508695:0:98971:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.513247:0:98971:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98971:1874531134076800:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.513292:0:98971:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x557:0x0](ffff9d85b3cf80c8) 00000080:00200000:2.0:1787695653.513300:0:98971:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x557:0x0], updating i_size 512 00000080:00200000:2.0:1787695653.522354:0:98972:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.522386:0:98972:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.522402:0:98972:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.522478:0:98972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.522489:0:98972:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.522499:0:98972:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98972 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.522526:0:98972:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98972:1874531134076928:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.522725:0:98972:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.527020:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134077056:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695653.527405:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134077312:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695653.530561:0:98972:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98972:1874531134076928:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.530588:0:98972:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530602:0:98972:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530615:0:98972:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530639:0:98972:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530658:0:98972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530668:0:98972:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530676:0:98972:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.530688:0:98972:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb3800 x1874531134076928/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.530703:0:98972:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530717:0:98972:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.530723:0:98972:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.530753:0:98972:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.530759:0:98972:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532922b 00000080:00010000:2.0:1787695653.530766:0:98972:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.530769:0:98972:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532922b from it ffffb5d4042b3c20 00010000:00010000:2.0:1787695653.530772:0:98972:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530782:0:98972:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e9532922b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1b8 expref: -99 pid: 98972 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.530809:0:98972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.530831:0:98972:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.530836:0:98972:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.530880:0:98972:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.530922:0:98972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329232 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.530930:0:98972:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329232 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.531231:0:98972:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329232 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98972 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.531256:0:98972:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98972:1874531134077440:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.531332:0:98972:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.539241:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134077056:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695653.539265:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134077312:192.168.201.137@tcp:400:kworker.0 00000080:00200000:2.0:1787695653.552464:0:98973:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.552499:0:98973:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.552572:0:98973:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98973 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.552584:0:98973:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98973 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.552594:0:98973:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98973 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.552750:0:98973:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98973:1874531134077824:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.553102:0:98973:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.557740:0:98973:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98973:1874531134077824:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.557769:0:98973:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.557782:0:98973:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.557794:0:98973:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.557805:0:98973:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.557818:0:98973:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.557826:0:98973:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.557834:0:98973:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.557846:0:98973:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb2d80 x1874531134077824/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.558274:0:98973:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.558289:0:98973:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.558297:0:98973:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.558328:0:98973:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.558334:0:98973:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329239 00000080:00010000:2.0:1787695653.558340:0:98973:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.558344:0:98973:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329239 from it ffffb5d4042a3be0 00010000:00010000:2.0:1787695653.558348:0:98973:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.558358:0:98973:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.558384:0:98973:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.558421:0:98973:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.558427:0:98973:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.558441:0:98973:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.558451:0:98973:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.558462:0:98973:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.558469:0:98973:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.558473:0:98973:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.558477:0:98973:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329239 00000080:00010000:2.0:1787695653.558481:0:98973:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329239 from it ffffb5d4042a3ce0 00010000:00010000:2.0:1787695653.558484:0:98973:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.558492:0:98973:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.558501:0:98973:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.558507:0:98973:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.558512:0:98973:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.558518:0:98973:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.558527:0:98973:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.558534:0:98973:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.558538:0:98973:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.558541:0:98973:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329239 00000080:00010000:2.0:1787695653.558544:0:98973:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329239 from it ffffb5d4042a3dc8 00010000:00010000:2.0:1787695653.558548:0:98973:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.558556:0:98973:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.558564:0:98973:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.558569:0:98973:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.558596:0:98973:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.558743:0:98973:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.558752:0:98973:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.558755:0:98973:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329239 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef1f7 expref: -99 pid: 98973 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.558780:0:98973:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.558789:0:98973:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb2d80 x1874531134078080/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.558811:0:98973:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98973:1874531134078080:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.559497:0:98973:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.566249:0:98973:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98973:1874531134078080:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.566302:0:98973:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x558:0x0](ffff9d85b3cf8908) 00000080:00200000:2.0:1787695653.566309:0:98973:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x558:0x0], updating i_size 512 00000080:00200000:2.0:1787695653.575587:0:98974:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.575624:0:98974:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.575642:0:98974:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.575715:0:98974:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98974 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.575726:0:98974:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98974 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.575743:0:98974:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98974 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.575775:0:98974:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98974:1874531134078208:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.575914:0:98974:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.579444:0:98974:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98974:1874531134078208:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.579472:0:98974:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579484:0:98974:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579497:0:98974:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579510:0:98974:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579523:0:98974:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579531:0:98974:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579539:0:98974:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.579555:0:98974:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb3100 x1874531134078208/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.579569:0:98974:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579585:0:98974:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.579592:0:98974:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.579683:0:98974:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.579691:0:98974:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329240 00000080:00010000:2.0:1787695653.579697:0:98974:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.579700:0:98974:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329240 from it ffffb5d40426bc20 00010000:00010000:2.0:1787695653.579704:0:98974:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579712:0:98974:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.579740:0:98974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.579763:0:98974:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.579768:0:98974:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.579789:0:98974:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.579831:0:98974:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98974 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.579838:0:98974:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98974 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.580195:0:98974:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98974 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.580380:0:98974:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98974:1874531134078336:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.580516:0:98974:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.583719:0:98974:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98974:1874531134078336:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.583748:0:98974:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.583769:0:98974:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x558:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.583777:0:98974:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.583789:0:98974:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.583800:0:98974:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.583813:0:98974:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.583820:0:98974:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.583833:0:98974:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.583846:0:98974:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb3100 x1874531134078336/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.584022:0:98974:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.584033:0:98974:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1368 (0 0) 00000080:00200000:2.0:1787695653.584040:0:98974:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695653.584132:0:98974:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x558:0x0](ffff9d85b3cf9148) 00000080:00200000:2.0:1787695653.584141:0:98974:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x558:0x0], updating i_size 512 00000080:00010000:2.0:1787695653.584148:0:98974:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x558:0x0](ffff9d85b3cf9148) for lock 0x678ae32e95329247 00000080:00010000:2.0:1787695653.584159:0:98974:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x558:0x0](ffff9d85b3cf9148) 00000080:00010000:2.0:1787695653.584163:0:98974:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329247 from it ffffb5d40426be08 00010000:00010000:2.0:1787695653.584167:0:98974:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.584177:0:98974:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.584209:0:98974:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.584226:0:98974:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.584237:0:98974:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.584253:0:98974:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.584260:0:98974:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.584265:0:98974:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.584268:0:98974:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329240 00000080:00010000:2.0:1787695653.584272:0:98974:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329240 from it ffffb5d40426bd90 00010000:00010000:2.0:1787695653.584274:0:98974:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.584281:0:98974:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.584290:0:98974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.584298:0:98974:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695653.584319:0:98974:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.584357:0:98974:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.584391:0:98974:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.584399:0:98974:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.584422:0:98974:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695653.584425:0:98974:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329247 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x558:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef22f expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.584442:0:98974:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e95329240 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef213 expref: -99 pid: 98974 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695653.584459:0:98974:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.584467:0:98974:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb3100 x1874531134078592/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.584489:0:98974:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98974:1874531134078592:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695653.584609:0:98974:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.593226:0:98974:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98974:1874531134078592:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695653.593302:0:98974:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x558:0x0](ffff9d85b3cf9148) 00000080:00200000:2.0:1787695653.652296:0:98977:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.652320:0:98977:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.652385:0:98977:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98977 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.652396:0:98977:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98977 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.652420:0:98977:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98977 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.652447:0:98977:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98977:1874531134079616:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.652583:0:98977:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.659080:0:98977:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98977:1874531134079616:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.659115:0:98977:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659129:0:98977:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659142:0:98977:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659153:0:98977:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659166:0:98977:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659173:0:98977:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659180:0:98977:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.659192:0:98977:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb2680 x1874531134079616/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.659206:0:98977:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659225:0:98977:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.659233:0:98977:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.659258:0:98977:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.659264:0:98977:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329263 00000080:00010000:2.0:1787695653.659270:0:98977:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.659273:0:98977:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329263 from it ffffb5d4042cbbe0 00010000:00010000:2.0:1787695653.659276:0:98977:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.659287:0:98977:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.659311:0:98977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.659333:0:98977:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.659338:0:98977:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.659352:0:98977:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.659362:0:98977:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.659373:0:98977:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.659381:0:98977:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.659385:0:98977:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.659388:0:98977:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329263 00000080:00010000:2.0:1787695653.659392:0:98977:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329263 from it ffffb5d4042cbce0 00010000:00010000:2.0:1787695653.659396:0:98977:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.659404:0:98977:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.659425:0:98977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.659431:0:98977:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.659437:0:98977:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.659443:0:98977:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.659451:0:98977:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.659458:0:98977:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.659462:0:98977:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.659465:0:98977:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329263 00000080:00010000:2.0:1787695653.659468:0:98977:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329263 from it ffffb5d4042cbdc8 00010000:00010000:2.0:1787695653.659471:0:98977:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.659479:0:98977:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.659487:0:98977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.659492:0:98977:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.659519:0:98977:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.659566:0:98977:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.659574:0:98977:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.659577:0:98977:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e95329263 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef2ad expref: -99 pid: 98977 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.659602:0:98977:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.659609:0:98977:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb2680 x1874531134079872/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.659632:0:98977:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98977:1874531134079872:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.659746:0:98977:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.664395:0:98977:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98977:1874531134079872:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.664460:0:98977:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x55a:0x0](ffff9d85b3cf9148) 00000080:00200000:2.0:1787695653.664468:0:98977:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x55a:0x0], updating i_size 512 00000080:00200000:2.0:1787695653.671769:0:98978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.671803:0:98978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.671820:0:98978:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.672359:0:98978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.672370:0:98978:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.672380:0:98978:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98978 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.672423:0:98978:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98978:1874531134080000:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.672558:0:98978:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.678725:0:98978:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98978:1874531134080000:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.678754:0:98978:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.678768:0:98978:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.678780:0:98978:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.678792:0:98978:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.678805:0:98978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.678814:0:98978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.678823:0:98978:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.678835:0:98978:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb1500 x1874531134080000/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.679040:0:98978:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.679057:0:98978:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.679066:0:98978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.679105:0:98978:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.679113:0:98978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532926a 00000080:00010000:2.0:1787695653.679119:0:98978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.679122:0:98978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532926a from it ffffb5d4042d3c20 00010000:00010000:2.0:1787695653.679126:0:98978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.679135:0:98978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.679163:0:98978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.679192:0:98978:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695653.679196:0:98978:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695653.679218:0:98978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695653.679264:0:98978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.679271:0:98978:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.679281:0:98978:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98978 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.679304:0:98978:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98978:1874531134080128:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.679383:0:98978:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.682508:0:98978:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98978:1874531134080128:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695653.682535:0:98978:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682557:0:98978:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682564:0:98978:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682574:0:98978:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682585:0:98978:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682597:0:98978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682605:0:98978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682612:0:98978:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.682642:0:98978:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb1500 x1874531134080128/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.682654:0:98978:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682665:0:98978:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1370 (0 0) 00000080:00200000:2.0:1787695653.682673:0:98978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695653.682707:0:98978:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x55a:0x0](ffff9d85b3cff448) 00000080:00200000:2.0:1787695653.682715:0:98978:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x55a:0x0], updating i_size 512 00000080:00010000:2.0:1787695653.682721:0:98978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x55a:0x0](ffff9d85b3cff448) for lock 0x678ae32e95329271 00000080:00010000:2.0:1787695653.682732:0:98978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x55a:0x0](ffff9d85b3cff448) 00000080:00010000:2.0:1787695653.682735:0:98978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329271 from it ffffb5d4042d3e08 00010000:00010000:2.0:1787695653.682739:0:98978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.682749:0:98978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.682779:0:98978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.682794:0:98978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.682804:0:98978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.682815:0:98978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.682827:0:98978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.682832:0:98978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695653.682835:0:98978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e9532926a 00000080:00010000:2.0:1787695653.682839:0:98978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532926a from it ffffb5d4042d3d90 00010000:00010000:2.0:1787695653.682843:0:98978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.682888:0:98978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.682898:0:98978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.682906:0:98978:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695653.682924:0:98978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.682974:0:98978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.683012:0:98978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.683020:0:98978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.683026:0:98978:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695653.683029:0:98978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e95329271 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x55a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef2e5 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.683053:0:98978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e9532926a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef2c9 expref: -99 pid: 98978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695653.683069:0:98978:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.683076:0:98978:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb1500 x1874531134080384/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.683097:0:98978:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98978:1874531134080384:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695653.683160:0:98978:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000780 going to sleep for 12 seconds 00000100:00100000:0.0:1787695653.703268:0:98979:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98979:1874531134080512:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695653.703301:0:98979:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703315:0:98979:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703326:0:98979:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703337:0:98979:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703350:0:98979:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703359:0:98979:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703367:0:98979:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695653.703378:0:98979:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2ca80 x1874531134080512/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695653.703392:0:98979:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703424:0:98979:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695653.703432:0:98979:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695653.703461:0:98979:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695653.703468:0:98979:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329278 00000080:00010000:0.0:1787695653.703473:0:98979:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.703476:0:98979:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329278 from it ffffb5d4042dbbe0 00010000:00010000:0.0:1787695653.703480:0:98979:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.703490:0:98979:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.703514:0:98979:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695653.703540:0:98979:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695653.703545:0:98979:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.703562:0:98979:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.703572:0:98979:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.703584:0:98979:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695653.703591:0:98979:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.703596:0:98979:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.703599:0:98979:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329278 00000080:00010000:0.0:1787695653.703602:0:98979:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329278 from it ffffb5d4042dbce0 00010000:00010000:0.0:1787695653.703606:0:98979:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695653.703613:0:98979:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.703622:0:98979:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.703710:0:98979:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695653.703717:0:98979:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.703724:0:98979:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.703733:0:98979:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695653.703740:0:98979:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695653.703743:0:98979:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695653.703746:0:98979:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329278 00000080:00010000:0.0:1787695653.703750:0:98979:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329278 from it ffffb5d4042dbdc8 00010000:00010000:0.0:1787695653.703754:0:98979:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695653.703762:0:98979:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695653.703771:0:98979:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695653.703777:0:98979:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695653.703821:0:98979:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e95329278 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef308 expref: -99 pid: 98979 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787695653.716434:0:98979:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3b100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98979:1874531134080768:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.716488:0:98979:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x55b:0x0](ffff9d85b3cfba88) 00000080:00200000:2.0:1787695653.716495:0:98979:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x55b:0x0], updating i_size 512 00000080:00200000:0.0:1787695653.725778:0:98980:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.725806:0:98980:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.725821:0:98980:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695653.726147:0:98980:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e9532927f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98980 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.726159:0:98980:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e9532927f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98980 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.726171:0:98980:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e9532927f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98980 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.726198:0:98980:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3aa00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98980:1874531134080896:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.726335:0:98980:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1600 going to sleep for 12 seconds 00000080:00200000:2.0:1787695653.765308:0:98981:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.765336:0:98981:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.766577:0:98981:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98981 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.766595:0:98981:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98981 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.766610:0:98981:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98981 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.766642:0:98981:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98981:1874531134081408:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695653.766755:0:98981:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.773247:0:98981:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98981:1874531134081408:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695653.773275:0:98981:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773288:0:98981:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773299:0:98981:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773310:0:98981:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773323:0:98981:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773332:0:98981:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773341:0:98981:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695653.773352:0:98981:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb3b80 x1874531134081408/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695670 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695653.773366:0:98981:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773381:0:98981:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695653.773387:0:98981:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695653.773429:0:98981:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695653.773436:0:98981:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532928d 00000080:00010000:2.0:1787695653.773443:0:98981:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.773446:0:98981:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532928d from it ffffb5d4042bbbe0 00010000:00010000:2.0:1787695653.773450:0:98981:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.773460:0:98981:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.773484:0:98981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695653.773506:0:98981:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695653.773511:0:98981:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.773524:0:98981:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.773533:0:98981:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.773545:0:98981:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695653.773552:0:98981:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.773557:0:98981:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.773560:0:98981:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532928d 00000080:00010000:2.0:1787695653.773564:0:98981:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532928d from it ffffb5d4042bbce0 00010000:00010000:2.0:1787695653.773567:0:98981:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695653.773575:0:98981:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.773584:0:98981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.773590:0:98981:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695653.773595:0:98981:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.773601:0:98981:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.773610:0:98981:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695653.773617:0:98981:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695653.773620:0:98981:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695653.773634:0:98981:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532928d 00000080:00010000:2.0:1787695653.773638:0:98981:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532928d from it ffffb5d4042bbdc8 00010000:00010000:2.0:1787695653.773642:0:98981:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695653.773650:0:98981:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695653.773659:0:98981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695653.773664:0:98981:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695653.773697:0:98981:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.773744:0:98981:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695653.773757:0:98981:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695653.773761:0:98981:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f600/0x678ae32e9532928d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef363 expref: -99 pid: 98981 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695653.773786:0:98981:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695653.773793:0:98981:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb3b80 x1874531134081664/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695653.773816:0:98981:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98981:1874531134081664:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695653.773926:0:98981:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.778378:0:98981:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98981:1874531134081664:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.778508:0:98981:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x55c:0x0](ffff9d85b3cf9988) 00000080:00200000:2.0:1787695653.778517:0:98981:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x55c:0x0], updating i_size 512 00000080:00200000:0.0:1787695653.796373:0:98982:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695653.796422:0:98982:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695653.796442:0:98982:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695653.796518:0:98982:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329294 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98982 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.796528:0:98982:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329294 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98982 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695653.796539:0:98982:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329294 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98982 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695653.796573:0:98982:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98982:1874531134081792:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695653.796761:0:98982:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1f00 going to sleep for 12 seconds 00000080:00200000:2.0:1787695653.886148:0:98984:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.886179:0:98984:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.886195:0:98984:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.886249:0:98984:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e953292a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98984 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.886258:0:98984:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e953292a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98984 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.886268:0:98984:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f600/0x678ae32e953292a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98984 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695653.886295:0:98984:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98984:1874531134082688:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695653.886381:0:98984:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000000 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.947359:0:98985:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9eae540 going to sleep for 12 seconds 00000100:00100000:2.0:1787695653.954196:0:98985:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98985:1874531134083456:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695653.954246:0:98985:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x55e:0x0](ffff9d85b3cfcb08) 00000080:00200000:2.0:1787695653.954253:0:98985:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x55e:0x0], updating i_size 512 00000080:00200000:2.0:1787695653.965745:0:98986:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695653.965773:0:98986:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695653.965787:0:98986:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695653.965843:0:98986:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953292be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98986 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.968740:0:98986:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953292be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98986 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695653.968763:0:98986:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ee00/0x678ae32e953292be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98986 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.006535:0:98986:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98986:1874531134083968:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695654.006584:0:98986:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x55e:0x0](ffff9d85b3cef448) 00000080:00200000:0.0:1787695654.030815:0:98987:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.030844:0:98987:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.030941:0:98987:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98987 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.030953:0:98987:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98987 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.030965:0:98987:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98987 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695654.030994:0:98987:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98987:1874531134084096:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695654.031089:0:98987:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.034541:0:98987:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98987:1874531134084096:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695654.034571:0:98987:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034584:0:98987:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034599:0:98987:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034611:0:98987:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034637:0:98987:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034646:0:98987:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034654:0:98987:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695654.034667:0:98987:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39500 x1874531134084096/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695654.034681:0:98987:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034698:0:98987:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695654.034706:0:98987:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695654.034740:0:98987:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695654.034747:0:98987:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953292cc 00000080:00010000:0.0:1787695654.034753:0:98987:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.034756:0:98987:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292cc from it ffffb5d40420bbe0 00010000:00010000:0.0:1787695654.034761:0:98987:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.034771:0:98987:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.034794:0:98987:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695654.034816:0:98987:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695654.034820:0:98987:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.034832:0:98987:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.034841:0:98987:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.035180:0:98987:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.035190:0:98987:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.035196:0:98987:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.035199:0:98987:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953292cc 00000080:00010000:0.0:1787695654.035203:0:98987:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292cc from it ffffb5d40420bce0 00010000:00010000:0.0:1787695654.035206:0:98987:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.035213:0:98987:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.035222:0:98987:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.035229:0:98987:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.035258:0:98987:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.035266:0:98987:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.035273:0:98987:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.035279:0:98987:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.035283:0:98987:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.035285:0:98987:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953292cc 00000080:00010000:0.0:1787695654.035288:0:98987:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292cc from it ffffb5d40420bdc8 00010000:00010000:0.0:1787695654.035291:0:98987:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.035297:0:98987:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695654.035306:0:98987:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.035312:0:98987:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695654.035350:0:98987:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695654.035410:0:98987:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695654.035417:0:98987:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695654.035420:0:98987:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e953292cc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef474 expref: -99 pid: 98987 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695654.035450:0:98987:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695654.035462:0:98987:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f39500 x1874531134084352/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695654.035490:0:98987:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98987:1874531134084352:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695654.035587:0:98987:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.046232:0:98987:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39500 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98987:1874531134084352:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695654.046419:0:98987:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x55f:0x0](ffff9d85b3cfd348) 00000080:00200000:2.0:1787695654.046432:0:98987:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x55f:0x0], updating i_size 512 00000100:00100000:2.0:1787695654.077339:0:98988:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98988:1874531134084480:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.077370:0:98988:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077384:0:98988:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077397:0:98988:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077422:0:98988:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077435:0:98988:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077444:0:98988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077453:0:98988:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.077467:0:98988:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2c000 x1874531134084480/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.077481:0:98988:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077501:0:98988:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.077511:0:98988:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.077566:0:98988:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.077574:0:98988:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953292d3 00000080:00010000:2.0:1787695654.077580:0:98988:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695654.077583:0:98988:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292d3 from it ffffb5d40425bc20 00010000:00010000:2.0:1787695654.077586:0:98988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077601:0:98988:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.077632:0:98988:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.077667:0:98988:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695654.077674:0:98988:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695654.077708:0:98988:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695654.077772:0:98988:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98988 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077781:0:98988:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98988 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.077792:0:98988:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98988 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.077817:0:98988:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98988:1874531134084608:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695654.077966:0:98988:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.085268:0:98988:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98988:1874531134084608:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.085300:0:98988:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085326:0:98988:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 4/1,0 mode: --/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085335:0:98988:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 4/1,0 mode: --/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085352:0:98988:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085362:0:98988:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085377:0:98988:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085384:0:98988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085391:0:98988:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.085412:0:98988:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb2300 x1874531134084608/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.085427:0:98988:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085439:0:98988:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1375 (0 0) 00000080:00200000:2.0:1787695654.085447:0:98988:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695654.085492:0:98988:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x55f:0x0](ffff9d85b3cfdb88) 00000080:00200000:2.0:1787695654.085503:0:98988:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x55f:0x0], updating i_size 512 00000080:00010000:2.0:1787695654.085512:0:98988:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x55f:0x0](ffff9d85b3cfdb88) for lock 0x678ae32e953292da 00000080:00010000:2.0:1787695654.085541:0:98988:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x55f:0x0](ffff9d85b3cfdb88) 00000080:00010000:2.0:1787695654.085548:0:98988:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292da from it ffffb5d40425be08 00010000:00010000:2.0:1787695654.085552:0:98988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.085562:0:98988:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.085607:0:98988:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.085638:0:98988:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.085652:0:98988:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.085665:0:98988:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.085673:0:98988:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.085678:0:98988:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695654.085681:0:98988:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953292d3 00000080:00010000:2.0:1787695654.085685:0:98988:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292d3 from it ffffb5d40425bd90 00010000:00010000:2.0:1787695654.085688:0:98988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.085695:0:98988:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.085704:0:98988:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.085711:0:98988:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695654.085733:0:98988:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.085769:0:98988:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.085808:0:98988:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 1/0,0 mode: --/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.085816:0:98988:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.085823:0:98988:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695654.085825:0:98988:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953292da lrc: 0/0,0 mode: --/PR res: [0x200000401:0x55f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef4ac expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.085843:0:98988:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031600/0x678ae32e953292d3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef490 expref: -99 pid: 98988 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695654.086180:0:98988:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.086189:0:98988:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb2300 x1874531134084864/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.086215:0:98988:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98988:1874531134084864:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695654.086303:0:98988:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.092671:0:98988:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98988:1874531134084864:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695654.092727:0:98988:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x55f:0x0](ffff9d85b3cfdb88) 00010000:00010000:0.0:1787695654.101467:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1787695654.193484:0:98991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.193513:0:98991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.193586:0:98991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.193598:0:98991:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.193608:0:98991:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98991 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.193636:0:98991:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98991:1874531134085888:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695654.193821:0:98991:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.199920:0:98991:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98991:1874531134085888:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695654.199964:0:98991:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.199977:0:98991:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.199990:0:98991:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.200003:0:98991:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.200017:0:98991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.200027:0:98991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.200041:0:98991:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.200053:0:98991:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb1c00 x1874531134085888/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.200067:0:98991:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.200081:0:98991:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.200089:0:98991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.200124:0:98991:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.200130:0:98991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953292f6 00000080:00010000:2.0:1787695654.200137:0:98991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.200140:0:98991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292f6 from it ffffb5d40425bbe0 00010000:00010000:2.0:1787695654.200144:0:98991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.200153:0:98991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.200178:0:98991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.200202:0:98991:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695654.200207:0:98991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.200221:0:98991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.200230:0:98991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.200247:0:98991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.200254:0:98991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.200258:0:98991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.200261:0:98991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953292f6 00000080:00010000:2.0:1787695654.200264:0:98991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292f6 from it ffffb5d40425bce0 00010000:00010000:2.0:1787695654.200267:0:98991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.200274:0:98991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.200282:0:98991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.200288:0:98991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.200293:0:98991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.200299:0:98991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.200307:0:98991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.200314:0:98991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.200318:0:98991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.200320:0:98991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953292f6 00000080:00010000:2.0:1787695654.200323:0:98991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292f6 from it ffffb5d40425bdc8 00010000:00010000:2.0:1787695654.200327:0:98991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.200334:0:98991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695654.200342:0:98991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.200348:0:98991:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695654.200385:0:98991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.200447:0:98991:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.200455:0:98991:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695654.200458:0:98991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e953292f6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef52a expref: -99 pid: 98991 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695654.200492:0:98991:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.200500:0:98991:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb1c00 x1874531134086144/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.200521:0:98991:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98991:1874531134086144:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695654.200618:0:98991:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.207075:0:98991:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98991:1874531134086144:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695654.207129:0:98991:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x561:0x0](ffff9d85b3cfdb88) 00000080:00200000:2.0:1787695654.207136:0:98991:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x561:0x0], updating i_size 512 00000080:00200000:0.0:1787695654.234295:0:98992:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x561:0x0](ffff9d85b3cd7448) 00000080:00200000:0.0:1787695654.234304:0:98992:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x561:0x0], updating i_size 512 00000080:00010000:0.0:1787695654.234312:0:98992:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x561:0x0](ffff9d85b3cd7448) for lock 0x678ae32e95329304 00000080:00010000:0.0:1787695654.234322:0:98992:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x561:0x0](ffff9d85b3cd7448) 00000080:00010000:0.0:1787695654.234326:0:98992:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329304 from it ffffb5d4042f3e08 00010000:00010000:0.0:1787695654.234331:0:98992:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e95329304 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x561:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef562 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.234341:0:98992:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e95329304 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x561:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef562 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.234375:0:98992:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695654.234393:0:98992:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.234416:0:98992:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e953292fd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef546 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.234428:0:98992:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e953292fd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef546 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.234441:0:98992:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.234446:0:98992:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695654.234449:0:98992:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953292fd 00000080:00010000:0.0:1787695654.234454:0:98992:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953292fd from it ffffb5d4042f3d90 00010000:00010000:0.0:1787695654.234457:0:98992:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e953292fd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef546 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.234465:0:98992:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e953292fd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef546 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.234473:0:98992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.234480:0:98992:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695654.234497:0:98992:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e953292fd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef546 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.234533:0:98992:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e95329304 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x561:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef562 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.234569:0:98992:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e95329304 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x561:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef562 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.234577:0:98992:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e953292fd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef546 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.234584:0:98992:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695654.234587:0:98992:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f800/0x678ae32e95329304 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x561:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef562 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.234608:0:98992:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030400/0x678ae32e953292fd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef546 expref: -99 pid: 98992 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695654.234635:0:98992:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695654.234643:0:98992:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f38700 x1874531134086656/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695654.234669:0:98992:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f38700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98992:1874531134086656:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695654.234880:0:98992:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.272552:0:98993:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98993:1874531134086784:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695654.272583:0:98993:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272597:0:98993:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272610:0:98993:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272621:0:98993:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272634:0:98993:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272642:0:98993:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272651:0:98993:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695654.272662:0:98993:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2e680 x1874531134086784/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695654.272677:0:98993:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272691:0:98993:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695654.272698:0:98993:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695654.272731:0:98993:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695654.272737:0:98993:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532930b 00000080:00010000:0.0:1787695654.272744:0:98993:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.272747:0:98993:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532930b from it ffffb5d4042fbbe0 00010000:00010000:0.0:1787695654.272751:0:98993:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.272760:0:98993:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.272787:0:98993:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695654.272811:0:98993:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695654.272818:0:98993:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.272834:0:98993:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.272843:0:98993:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.272907:0:98993:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.272915:0:98993:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.272920:0:98993:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.272989:0:98993:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532930b 00000080:00010000:0.0:1787695654.272996:0:98993:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532930b from it ffffb5d4042fbce0 00010000:00010000:0.0:1787695654.273000:0:98993:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.273009:0:98993:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.273018:0:98993:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.273025:0:98993:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.273033:0:98993:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.273040:0:98993:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.273049:0:98993:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.273056:0:98993:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.273060:0:98993:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.273063:0:98993:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532930b 00000080:00010000:0.0:1787695654.273067:0:98993:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532930b from it ffffb5d4042fbdc8 00010000:00010000:0.0:1787695654.273071:0:98993:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.273078:0:98993:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695654.273086:0:98993:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.273091:0:98993:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695654.273130:0:98993:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695654.273182:0:98993:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695654.273190:0:98993:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695654.273193:0:98993:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580030a00/0x678ae32e9532930b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef585 expref: -99 pid: 98993 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695654.273221:0:98993:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695654.273229:0:98993:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f3aa00 x1874531134087040/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695654.273251:0:98993:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f3aa00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98993:1874531134087040:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695654.273334:0:98993:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.279736:0:98993:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f3aa00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98993:1874531134087040:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695654.279790:0:98993:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x562:0x0](ffff9d85b3cd7448) 00000080:00200000:0.0:1787695654.279799:0:98993:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x562:0x0], updating i_size 512 00000080:00200000:2.0:1787695654.295793:0:98994:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.295827:0:98994:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.295842:0:98994:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695654.295944:0:98994:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98994 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.295955:0:98994:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98994 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.295966:0:98994:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98994 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.295994:0:98994:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98994:1874531134087168:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695654.296087:0:98994:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.300500:0:98994:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98994:1874531134087168:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.300532:0:98994:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300546:0:98994:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300561:0:98994:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300574:0:98994:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300588:0:98994:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300602:0:98994:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300611:0:98994:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.300637:0:98994:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb0700 x1874531134087168/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.300662:0:98994:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300679:0:98994:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.300688:0:98994:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.300720:0:98994:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.300727:0:98994:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329312 00000080:00010000:2.0:1787695654.300733:0:98994:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695654.300737:0:98994:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329312 from it ffffb5d40420bc20 00010000:00010000:2.0:1787695654.300740:0:98994:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300750:0:98994:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.300778:0:98994:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.300805:0:98994:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695654.300811:0:98994:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695654.300831:0:98994:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695654.300910:0:98994:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 98994 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300918:0:98994:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98994 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.300938:0:98994:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 98994 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.300962:0:98994:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98994:1874531134087296:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695654.301039:0:98994:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.306939:0:98994:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98994:1874531134087296:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.306967:0:98994:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.306992:0:98994:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x562:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307000:0:98994:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307011:0:98994:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307023:0:98994:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307036:0:98994:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307046:0:98994:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307055:0:98994:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.307279:0:98994:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb0700 x1874531134087296/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.307293:0:98994:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307304:0:98994:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1378 (0 0) 00000080:00200000:2.0:1787695654.307311:0:98994:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695654.307344:0:98994:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x562:0x0](ffff9d85b3cfec08) 00000080:00200000:2.0:1787695654.307353:0:98994:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x562:0x0], updating i_size 512 00000080:00010000:2.0:1787695654.307359:0:98994:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x562:0x0](ffff9d85b3cfec08) for lock 0x678ae32e95329319 00000080:00010000:2.0:1787695654.307370:0:98994:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x562:0x0](ffff9d85b3cfec08) 00000080:00010000:2.0:1787695654.307374:0:98994:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329319 from it ffffb5d40420be08 00010000:00010000:2.0:1787695654.307377:0:98994:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.307386:0:98994:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.307428:0:98994:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.307444:0:98994:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.307454:0:98994:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.307466:0:98994:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.307473:0:98994:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.307478:0:98994:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695654.307481:0:98994:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329312 00000080:00010000:2.0:1787695654.307485:0:98994:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329312 from it ffffb5d40420bd90 00010000:00010000:2.0:1787695654.307489:0:98994:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.307497:0:98994:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.307506:0:98994:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.307513:0:98994:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695654.307534:0:98994:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.307566:0:98994:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.307599:0:98994:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.307607:0:98994:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.307617:0:98994:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695654.307621:0:98994:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329319 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x562:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef5bd expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.307794:0:98994:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329312 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef5a1 expref: -99 pid: 98994 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695654.307811:0:98994:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.307819:0:98994:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb0700 x1874531134087552/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.307842:0:98994:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98994:1874531134087552:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695654.308093:0:98994:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.329793:0:98994:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98994:1874531134087552:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695654.329906:0:98994:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x562:0x0](ffff9d85b3cfec08) 00000080:00200000:2.0:1787695654.370499:0:98996:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.370531:0:98996:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.370547:0:98996:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695654.370603:0:98996:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329327 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 98996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.370612:0:98996:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329327 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.370622:0:98996:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329327 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 98996 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.370648:0:98996:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:98996:1874531134088064:192.168.201.137@tcp:101:rmdir.0 00000080:00200000:2.0:1787695654.443811:0:98997:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.443839:0:98997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.444697:0:98997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 98997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.444714:0:98997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.444727:0:98997:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 98997 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.444763:0:98997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98997:1874531134088576:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695654.447565:0:98997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.452770:0:98997:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98997:1874531134088576:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695654.452798:0:98997:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.452812:0:98997:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.452824:0:98997:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.452836:0:98997:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.452887:0:98997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.452903:0:98997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.452911:0:98997:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.453155:0:98997:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb3800 x1874531134088576/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.453174:0:98997:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.453188:0:98997:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.453195:0:98997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.453225:0:98997:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.453230:0:98997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329335 00000080:00010000:2.0:1787695654.453236:0:98997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.453239:0:98997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329335 from it ffffb5d4042fbbe0 00010000:00010000:2.0:1787695654.453243:0:98997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.453252:0:98997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.453277:0:98997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.453298:0:98997:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695654.453303:0:98997:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.453315:0:98997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.453324:0:98997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.453335:0:98997:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.453341:0:98997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.453346:0:98997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.453349:0:98997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329335 00000080:00010000:2.0:1787695654.453352:0:98997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329335 from it ffffb5d4042fbce0 00010000:00010000:2.0:1787695654.453354:0:98997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.453361:0:98997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.453369:0:98997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.453375:0:98997:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.453381:0:98997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.453387:0:98997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.453406:0:98997:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.453413:0:98997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.453417:0:98997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.453420:0:98997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329335 00000080:00010000:2.0:1787695654.453424:0:98997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329335 from it ffffb5d4042fbdc8 00010000:00010000:2.0:1787695654.453427:0:98997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.453433:0:98997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695654.453440:0:98997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.453445:0:98997:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695654.453478:0:98997:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.453525:0:98997:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.453533:0:98997:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695654.453536:0:98997:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329335 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef63b expref: -99 pid: 98997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695654.453563:0:98997:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.453570:0:98997:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb3800 x1874531134088832/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.453593:0:98997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98997:1874531134088832:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695654.453685:0:98997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.458291:0:98997:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:98997:1874531134088832:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695654.458343:0:98997:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x564:0x0](ffff9d85b3cfec08) 00000080:00200000:2.0:1787695654.458350:0:98997:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x564:0x0], updating i_size 512 00000080:00200000:0.0:1787695654.600372:0:99000:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695654.605717:0:99000:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.605738:0:99000:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695654.605798:0:99000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 99000 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.605807:0:99000:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99000 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.605817:0:99000:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99000 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695654.606277:0:99000:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99000:1874531134089856:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695654.606447:0:99000:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.615816:0:99000:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99000:1874531134089856:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.615845:0:99000:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616046:0:99000:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616125:0:99000:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616138:0:99000:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616153:0:99000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616236:0:99000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616243:0:99000:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.616254:0:99000:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39c00 x1874531134089856/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.616267:0:99000:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616281:0:99000:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.616290:0:99000:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.616317:0:99000:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.616323:0:99000:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329351 00000080:00010000:2.0:1787695654.616329:0:99000:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695654.616332:0:99000:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329351 from it ffffb5d404273c20 00010000:00010000:2.0:1787695654.616335:0:99000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616345:0:99000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.616370:0:99000:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.616404:0:99000:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695654.616409:0:99000:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695654.616427:0:99000:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695654.616469:0:99000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 99000 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616477:0:99000:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99000 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.616485:0:99000:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99000 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.616507:0:99000:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99000:1874531134089984:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695654.616578:0:99000:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.625325:0:99000:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99000:1874531134089984:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.625353:0:99000:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625379:0:99000:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x565:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625387:0:99000:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625411:0:99000:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625421:0:99000:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625435:0:99000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625443:0:99000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625451:0:99000:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.625464:0:99000:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85c0fb2a00 x1874531134089984/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.625478:0:99000:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625493:0:99000:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1381 (0 0) 00000080:00200000:2.0:1787695654.625499:0:99000:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695654.625530:0:99000:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x565:0x0](ffff9d85b3cfb248) 00000080:00200000:2.0:1787695654.625538:0:99000:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x565:0x0], updating i_size 512 00000080:00010000:2.0:1787695654.625545:0:99000:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x565:0x0](ffff9d85b3cfb248) for lock 0x678ae32e95329358 00000080:00010000:2.0:1787695654.625558:0:99000:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x565:0x0](ffff9d85b3cfb248) 00000080:00010000:2.0:1787695654.625561:0:99000:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329358 from it ffffb5d404273e08 00010000:00010000:2.0:1787695654.625565:0:99000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.625573:0:99000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.625603:0:99000:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.625618:0:99000:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.626253:0:99000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.626269:0:99000:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.626277:0:99000:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.626283:0:99000:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695654.626286:0:99000:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329351 00000080:00010000:2.0:1787695654.626292:0:99000:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329351 from it ffffb5d404273d90 00010000:00010000:2.0:1787695654.626295:0:99000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.626304:0:99000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.626315:0:99000:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.626324:0:99000:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695654.626346:0:99000:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.626377:0:99000:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.626421:0:99000:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.626430:0:99000:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.626435:0:99000:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695654.626438:0:99000:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e95329358 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x565:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef6ce expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.626455:0:99000:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ce00/0x678ae32e95329351 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef6b2 expref: -99 pid: 99000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695654.626470:0:99000:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.626478:0:99000:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85c0fb2a00 x1874531134090240/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.626505:0:99000:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99000:1874531134090240:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695654.626589:0:99000:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.646280:0:99000:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99000:1874531134090240:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695654.646325:0:99000:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x565:0x0](ffff9d85b3cfb248) 00000080:00200000:0.0:1787695654.661550:0:99001:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.661576:0:99001:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.662176:0:99001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.662187:0:99001:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.662198:0:99001:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99001 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695654.662224:0:99001:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99001:1874531134090368:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695654.662375:0:99001:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.670590:0:99001:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99001:1874531134090368:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695654.670621:0:99001:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670638:0:99001:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670650:0:99001:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670663:0:99001:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670677:0:99001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670686:0:99001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670695:0:99001:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695654.670707:0:99001:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9f39c00 x1874531134090368/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695654.670720:0:99001:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670733:0:99001:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695654.670740:0:99001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695654.670768:0:99001:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695654.670774:0:99001:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532935f 00000080:00010000:0.0:1787695654.670780:0:99001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.670783:0:99001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532935f from it ffffb5d40421bbe0 00010000:00010000:0.0:1787695654.670787:0:99001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.670796:0:99001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.670818:0:99001:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695654.670871:0:99001:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695654.670876:0:99001:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.670889:0:99001:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.670898:0:99001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.670910:0:99001:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.670917:0:99001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.671033:0:99001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.671038:0:99001:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532935f 00000080:00010000:0.0:1787695654.671043:0:99001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532935f from it ffffb5d40421bce0 00010000:00010000:0.0:1787695654.671046:0:99001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.671125:0:99001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.671140:0:99001:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.671147:0:99001:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695654.671155:0:99001:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.671162:0:99001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.671170:0:99001:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.671176:0:99001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.671180:0:99001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695654.671183:0:99001:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532935f 00000080:00010000:0.0:1787695654.671187:0:99001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532935f from it ffffb5d40421bdc8 00010000:00010000:0.0:1787695654.671190:0:99001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.671198:0:99001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695654.671206:0:99001:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.671211:0:99001:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695654.671243:0:99001:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695654.671285:0:99001:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695654.671297:0:99001:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695654.671300:0:99001:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532935f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef6f1 expref: -99 pid: 99001 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695654.671325:0:99001:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695654.671333:0:99001:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9f39c00 x1874531134090624/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695654.671354:0:99001:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99001:1874531134090624:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695654.671438:0:99001:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.683226:0:99001:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9f39c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99001:1874531134090624:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695654.683284:0:99001:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x566:0x0](ffff9d85b3cd0908) 00000080:00200000:0.0:1787695654.683291:0:99001:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x566:0x0], updating i_size 512 00000080:00200000:2.0:1787695654.733382:0:99003:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.733421:0:99003:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.733483:0:99003:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99003 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.733494:0:99003:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99003 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.733505:0:99003:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99003 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.733534:0:99003:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fc700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99003:1874531134091264:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695654.733683:0:99003:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.737606:0:99003:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fc700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99003:1874531134091264:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695654.737710:0:99003:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.737726:0:99003:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.737738:0:99003:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.737756:0:99003:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.737768:0:99003:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.737778:0:99003:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.737786:0:99003:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.737799:0:99003:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fc700 x1874531134091264/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.737813:0:99003:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.737827:0:99003:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.737834:0:99003:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.738154:0:99003:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.738162:0:99003:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329374 00000080:00010000:2.0:1787695654.738168:0:99003:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.738171:0:99003:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329374 from it ffffb5d404313be0 00010000:00010000:2.0:1787695654.738174:0:99003:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.738189:0:99003:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.738215:0:99003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.738236:0:99003:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695654.738240:0:99003:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.738252:0:99003:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.738261:0:99003:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.738272:0:99003:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.738279:0:99003:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.738283:0:99003:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.738286:0:99003:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329374 00000080:00010000:2.0:1787695654.738290:0:99003:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329374 from it ffffb5d404313ce0 00010000:00010000:2.0:1787695654.738293:0:99003:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.738300:0:99003:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.738309:0:99003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.738315:0:99003:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.738321:0:99003:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.738326:0:99003:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.738333:0:99003:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.738340:0:99003:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.738343:0:99003:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.738345:0:99003:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329374 00000080:00010000:2.0:1787695654.738349:0:99003:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329374 from it ffffb5d404313dc8 00010000:00010000:2.0:1787695654.738352:0:99003:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.738359:0:99003:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695654.738367:0:99003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.738372:0:99003:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695654.738414:0:99003:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.738467:0:99003:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.738474:0:99003:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695654.738477:0:99003:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329374 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef74c expref: -99 pid: 99003 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695654.738504:0:99003:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.738513:0:99003:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fc700 x1874531134091520/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.738536:0:99003:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fc700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99003:1874531134091520:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695654.738677:0:99003:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.743462:0:99003:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fc700 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99003:1874531134091520:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695654.743510:0:99003:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x567:0x0](ffff9d85b3cfb248) 00000080:00200000:2.0:1787695654.743518:0:99003:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x567:0x0], updating i_size 512 00000080:00200000:2.0:1787695654.751526:0:99004:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.751553:0:99004:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.751567:0:99004:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695654.751747:0:99004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 99004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.751757:0:99004:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.751769:0:99004:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99004 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.751796:0:99004:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99004:1874531134091648:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695654.752568:0:99004:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.756380:0:99004:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99004:1874531134091648:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.756490:0:99004:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.756504:0:99004:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.756522:0:99004:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.756535:0:99004:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.756549:0:99004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.756559:0:99004:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.756568:0:99004:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.756581:0:99004:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fe680 x1874531134091648/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.756595:0:99004:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e9532937b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef768 expref: -99 pid: 99004 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.756608:0:99004:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.756616:0:99004:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.794285:0:99005:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.794315:0:99005:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.794382:0:99005:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99005 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.794428:0:99005:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99005 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.794439:0:99005:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99005 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.794466:0:99005:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99005:1874531134092160:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695654.794560:0:99005:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.800645:0:99005:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99005:1874531134092160:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695654.800675:0:99005:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800688:0:99005:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800700:0:99005:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800710:0:99005:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800722:0:99005:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800731:0:99005:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800743:0:99005:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.800755:0:99005:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b72fe680 x1874531134092160/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.800769:0:99005:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800782:0:99005:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.800789:0:99005:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.800815:0:99005:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.800821:0:99005:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329389 00000080:00010000:2.0:1787695654.800826:0:99005:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.800829:0:99005:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329389 from it ffffb5d404323be0 00010000:00010000:2.0:1787695654.800832:0:99005:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.800871:0:99005:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.800895:0:99005:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.800925:0:99005:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695654.800930:0:99005:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.800944:0:99005:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.800953:0:99005:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.800970:0:99005:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.800977:0:99005:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.800981:0:99005:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.800984:0:99005:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329389 00000080:00010000:2.0:1787695654.800988:0:99005:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329389 from it ffffb5d404323ce0 00010000:00010000:2.0:1787695654.800991:0:99005:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.800999:0:99005:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.801008:0:99005:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.801014:0:99005:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.801019:0:99005:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.801025:0:99005:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.801033:0:99005:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.801040:0:99005:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.801043:0:99005:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.801046:0:99005:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329389 00000080:00010000:2.0:1787695654.801049:0:99005:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329389 from it ffffb5d404323dc8 00010000:00010000:2.0:1787695654.801053:0:99005:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.801061:0:99005:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695654.801069:0:99005:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.801074:0:99005:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695654.801103:0:99005:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.801142:0:99005:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.801149:0:99005:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695654.801152:0:99005:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e95329389 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef7a7 expref: -99 pid: 99005 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695654.801175:0:99005:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.801182:0:99005:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b72fe680 x1874531134092416/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.801204:0:99005:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99005:1874531134092416:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695654.801284:0:99005:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.805284:0:99005:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b72fe680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99005:1874531134092416:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695654.805334:0:99005:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x568:0x0](ffff9d85b3cfc2c8) 00000080:00200000:2.0:1787695654.805341:0:99005:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x568:0x0], updating i_size 512 00000080:00200000:0.0:1787695654.830510:0:99006:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695654.830536:0:99006:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.830550:0:99006:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695654.830604:0:99006:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 99006 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.830614:0:99006:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99006 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.830624:0:99006:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99006 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695654.830649:0:99006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99006:1874531134092544:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695654.830882:0:99006:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.854191:0:99006:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85879abd80 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.857127:0:99006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99006:1874531134092672:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695654.857158:0:99006:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857182:0:99006:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x568:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857190:0:99006:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857201:0:99006:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857213:0:99006:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857226:0:99006:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857233:0:99006:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857239:0:99006:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.857251:0:99006:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d2c700 x1874531134092672/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.857263:0:99006:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857274:0:99006:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1384 (0 0) 00000080:00200000:2.0:1787695654.857283:0:99006:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695654.857321:0:99006:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x568:0x0](ffff9d85b3cfaa08) 00000080:00200000:2.0:1787695654.857330:0:99006:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x568:0x0], updating i_size 512 00000080:00010000:2.0:1787695654.857337:0:99006:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x568:0x0](ffff9d85b3cfaa08) for lock 0x678ae32e95329397 00000080:00010000:2.0:1787695654.857348:0:99006:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x568:0x0](ffff9d85b3cfaa08) 00000080:00010000:2.0:1787695654.857352:0:99006:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329397 from it ffffb5d40432be08 00010000:00010000:2.0:1787695654.857355:0:99006:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.857364:0:99006:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.857408:0:99006:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.857424:0:99006:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.857433:0:99006:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7c3 expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.857444:0:99006:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7c3 expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.857450:0:99006:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.857455:0:99006:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695654.857458:0:99006:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329390 00000080:00010000:2.0:1787695654.857462:0:99006:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329390 from it ffffb5d40432bd90 00010000:00010000:2.0:1787695654.857465:0:99006:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef7c3 expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.857472:0:99006:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef7c3 expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.857480:0:99006:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.857487:0:99006:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695654.857505:0:99006:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef7c3 expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.857538:0:99006:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.857602:0:99006:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.857612:0:99006:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef7c3 expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.857618:0:99006:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695654.857621:0:99006:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580030e00/0x678ae32e95329397 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x568:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef7df expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.857640:0:99006:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c600/0x678ae32e95329390 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef7c3 expref: -99 pid: 99006 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695654.857657:0:99006:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.857664:0:99006:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd4380 x1874531134092928/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.857693:0:99006:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99006:1874531134092928:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695654.857784:0:99006:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.862475:0:99006:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99006:1874531134092928:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695654.862527:0:99006:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x568:0x0](ffff9d85b3cfaa08) 00010000:00010000:2.0:1787695654.875533:0:99007:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99007 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.875549:0:99007:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99007 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.875559:0:99007:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99007 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.875586:0:99007:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99007:1874531134093056:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695654.876491:0:99007:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.888700:0:99007:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99007:1874531134093056:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695654.888728:0:99007:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.888741:0:99007:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.888754:0:99007:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.888767:0:99007:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.888781:0:99007:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.888790:0:99007:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.888798:0:99007:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.888809:0:99007:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce2300 x1874531134093056/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.888824:0:99007:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.889022:0:99007:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.889031:0:99007:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.889060:0:99007:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.889067:0:99007:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532939e 00000080:00010000:2.0:1787695654.889073:0:99007:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.889076:0:99007:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532939e from it ffffb5d40421bbe0 00010000:00010000:2.0:1787695654.889080:0:99007:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.889089:0:99007:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.889118:0:99007:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.889140:0:99007:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695654.889145:0:99007:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.889157:0:99007:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.889166:0:99007:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.889176:0:99007:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.889183:0:99007:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.889187:0:99007:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.889190:0:99007:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532939e 00000080:00010000:2.0:1787695654.889193:0:99007:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532939e from it ffffb5d40421bce0 00010000:00010000:2.0:1787695654.889197:0:99007:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.889205:0:99007:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.889213:0:99007:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.889219:0:99007:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.889225:0:99007:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.889231:0:99007:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.889245:0:99007:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.889253:0:99007:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.889256:0:99007:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.889259:0:99007:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532939e 00000080:00010000:2.0:1787695654.889263:0:99007:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532939e from it ffffb5d40421bdc8 00010000:00010000:2.0:1787695654.889266:0:99007:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.889273:0:99007:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695654.889281:0:99007:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.889286:0:99007:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695654.889314:0:99007:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.889352:0:99007:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.889360:0:99007:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695654.889364:0:99007:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e9532939e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef802 expref: -99 pid: 99007 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695654.889401:0:99007:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.889409:0:99007:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd5180 x1874531134093312/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.889431:0:99007:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99007:1874531134093312:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695654.889566:0:99007:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.897646:0:99007:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99007:1874531134093312:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695654.897703:0:99007:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x569:0x0](ffff9d85b3cfaa08) 00000080:00200000:2.0:1787695654.897711:0:99007:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x569:0x0], updating i_size 512 00000100:00100000:2.0:1787695654.929227:0:99008:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99008:1874531134093824:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695654.929269:0:99008:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x569:0x0](ffff9d85b3d042c8) 00000080:00200000:2.0:1787695654.938520:0:99009:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.938548:0:99009:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.938619:0:99009:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99009 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.938718:0:99009:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99009 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.938732:0:99009:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99009 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.938761:0:99009:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99009:1874531134093952:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695654.939141:0:99009:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.942484:0:99009:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99009:1874531134093952:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695654.942524:0:99009:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942539:0:99009:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942559:0:99009:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942571:0:99009:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942586:0:99009:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942594:0:99009:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942601:0:99009:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695654.942613:0:99009:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd4a80 x1874531134093952/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695654.942699:0:99009:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942717:0:99009:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695654.942724:0:99009:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695654.942754:0:99009:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695654.942761:0:99009:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293b3 00000080:00010000:2.0:1787695654.942766:0:99009:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.942770:0:99009:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293b3 from it ffffb5d40433bbe0 00010000:00010000:2.0:1787695654.942774:0:99009:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.942784:0:99009:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.942811:0:99009:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695654.942834:0:99009:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695654.943079:0:99009:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.943095:0:99009:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.943105:0:99009:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.943118:0:99009:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695654.943125:0:99009:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.943130:0:99009:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.943134:0:99009:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293b3 00000080:00010000:2.0:1787695654.943138:0:99009:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293b3 from it ffffb5d40433bce0 00010000:00010000:2.0:1787695654.943141:0:99009:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695654.943149:0:99009:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.943158:0:99009:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.943165:0:99009:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695654.943171:0:99009:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.943176:0:99009:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.943184:0:99009:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695654.943190:0:99009:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695654.943194:0:99009:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695654.943196:0:99009:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293b3 00000080:00010000:2.0:1787695654.943199:0:99009:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293b3 from it ffffb5d40433bdc8 00010000:00010000:2.0:1787695654.943203:0:99009:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695654.943210:0:99009:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695654.943218:0:99009:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695654.943224:0:99009:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695654.943258:0:99009:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.943316:0:99009:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695654.943323:0:99009:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695654.943327:0:99009:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e200/0x678ae32e953293b3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef85d expref: -99 pid: 99009 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695654.943363:0:99009:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695654.943373:0:99009:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd4a80 x1874531134094208/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695654.943408:0:99009:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99009:1874531134094208:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695654.943551:0:99009:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695654.949088:0:99009:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99009:1874531134094208:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:2.0:1787695654.949148:0:99009:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x56a:0x0](ffff9d85b3cfe3c8) 00000080:00200000:2.0:1787695654.949156:0:99009:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x56a:0x0], updating i_size 512 00000080:00200000:2.0:1787695654.958169:0:99010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695654.958206:0:99010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695654.958223:0:99010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1787695654.958293:0:99010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 99010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.958302:0:99010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695654.958313:0:99010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695654.958344:0:99010:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99010:1874531134094336:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695654.962214:0:99010:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000480 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.966077:0:99010:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99010:1874531134094336:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695654.966117:0:99010:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966131:0:99010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966144:0:99010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966157:0:99010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966171:0:99010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966178:0:99010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966186:0:99010:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695654.966199:0:99010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd5880 x1874531134094336/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695654.966213:0:99010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966233:0:99010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695654.966242:0:99010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695654.966280:0:99010:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695654.966287:0:99010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953293ba 00000080:00010000:0.0:1787695654.966292:0:99010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695654.966296:0:99010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293ba from it ffffb5d404343c20 00010000:00010000:0.0:1787695654.966300:0:99010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966311:0:99010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.966342:0:99010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695654.966374:0:99010:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695654.966380:0:99010:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695654.966416:0:99010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695654.966469:0:99010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 99010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966477:0:99010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.966486:0:99010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695654.966510:0:99010:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99010:1874531134094464:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695654.966606:0:99010:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.976904:0:99010:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99010:1874531134094464:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695654.976940:0:99010:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.976963:0:99010:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.976972:0:99010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.976984:0:99010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.976995:0:99010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.977010:0:99010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.977018:0:99010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.977026:0:99010:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695654.977038:0:99010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d0ad80 x1874531134094464/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695671 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695654.977051:0:99010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.977063:0:99010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1386 (0 0) 00000080:00200000:0.0:1787695654.977072:0:99010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695654.977104:0:99010:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x56a:0x0](ffff9d85b3cd42c8) 00000080:00200000:0.0:1787695654.977113:0:99010:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x56a:0x0], updating i_size 512 00000080:00010000:0.0:1787695654.977120:0:99010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x56a:0x0](ffff9d85b3cd42c8) for lock 0x678ae32e953293c1 00000080:00010000:0.0:1787695654.977131:0:99010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x56a:0x0](ffff9d85b3cd42c8) 00000080:00010000:0.0:1787695654.977135:0:99010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293c1 from it ffffb5d404343e08 00010000:00010000:0.0:1787695654.977138:0:99010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695654.977147:0:99010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.977178:0:99010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695654.977194:0:99010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695654.977204:0:99010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.977215:0:99010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695654.977222:0:99010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695654.977226:0:99010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695654.977229:0:99010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953293ba 00000080:00010000:0.0:1787695654.977232:0:99010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293ba from it ffffb5d404343d90 00010000:00010000:0.0:1787695654.977235:0:99010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.977242:0:99010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695654.977250:0:99010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695654.977257:0:99010:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695654.977277:0:99010:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.977310:0:99010:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.977348:0:99010:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.977355:0:99010:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695654.977361:0:99010:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695654.977364:0:99010:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293c1 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x56a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef895 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695654.977381:0:99010:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e200/0x678ae32e953293ba lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef879 expref: -99 pid: 99010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695654.977405:0:99010:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695654.977411:0:99010:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d0ad80 x1874531134094720/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695654.977439:0:99010:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99010:1874531134094720:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695654.977511:0:99010:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:0.0:1787695654.990578:0:99010:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99010:1874531134094720:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695654.990641:0:99010:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x56a:0x0](ffff9d85b3cd42c8) 00000080:00200000:0.0:1787695655.030650:0:99012:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695655.030678:0:99012:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.030691:0:99012:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695655.030746:0:99012:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 99012 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.030756:0:99012:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99012 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.030766:0:99012:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99012 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695655.030792:0:99012:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99012:1874531134095232:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695655.031032:0:99012:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1a80 going to sleep for 12 seconds 00000100:00100000:2.0:1787695655.034306:0:99012:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99012:1874531134095232:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695655.034337:0:99012:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034359:0:99012:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034372:0:99012:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034394:0:99012:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034409:0:99012:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034418:0:99012:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034426:0:99012:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695655.034437:0:99012:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d08000 x1874531134095232/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695655.034451:0:99012:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034464:0:99012:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695655.034470:0:99012:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695655.034500:0:99012:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695655.034508:0:99012:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953293cf 00000080:00010000:2.0:1787695655.034514:0:99012:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695655.034517:0:99012:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293cf from it ffffb5d404313c20 00010000:00010000:2.0:1787695655.034520:0:99012:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034531:0:99012:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695655.034561:0:99012:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695655.034585:0:99012:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695655.034590:0:99012:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695655.034612:0:99012:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695655.034660:0:99012:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 99012 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034670:0:99012:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99012 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.034681:0:99012:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99012 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695655.034704:0:99012:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99012:1874531134095360:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695655.034787:0:99012:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695655.045564:0:99012:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99012:1874531134095360:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695655.045592:0:99012:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045620:0:99012:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045640:0:99012:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045652:0:99012:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045663:0:99012:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045675:0:99012:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045684:0:99012:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045692:0:99012:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695655.045705:0:99012:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd7800 x1874531134095360/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695655.045718:0:99012:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045734:0:99012:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1387 (0 0) 00000080:00200000:2.0:1787695655.045742:0:99012:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695655.045781:0:99012:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x56b:0x0](ffff9d85b3cff448) 00000080:00200000:2.0:1787695655.045792:0:99012:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x56b:0x0], updating i_size 512 00000080:00010000:2.0:1787695655.045801:0:99012:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x56b:0x0](ffff9d85b3cff448) for lock 0x678ae32e953293d6 00000080:00010000:2.0:1787695655.045816:0:99012:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x56b:0x0](ffff9d85b3cff448) 00000080:00010000:2.0:1787695655.045821:0:99012:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293d6 from it ffffb5d404313e08 00010000:00010000:2.0:1787695655.045825:0:99012:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.045847:0:99012:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695655.045949:0:99012:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695655.045966:0:99012:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695655.045978:0:99012:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695655.045990:0:99012:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695655.045998:0:99012:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695655.046002:0:99012:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695655.046006:0:99012:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953293cf 00000080:00010000:2.0:1787695655.046010:0:99012:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293cf from it ffffb5d404313d90 00010000:00010000:2.0:1787695655.046014:0:99012:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695655.046022:0:99012:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695655.046032:0:99012:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695655.046040:0:99012:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:2.0:1787695655.046066:0:99012:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695655.046103:0:99012:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695655.046141:0:99012:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695655.046148:0:99012:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695655.046154:0:99012:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695655.046158:0:99012:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e953293d6 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x56b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef8f0 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695655.046174:0:99012:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898ca00/0x678ae32e953293cf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef8d4 expref: -99 pid: 99012 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1787695655.046188:0:99012:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695655.046195:0:99012:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd7800 x1874531134095616/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695655.046220:0:99012:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99012:1874531134095616:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:2.0:1787695655.046431:0:99012:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695655.051031:0:99012:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99012:1874531134095616:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:2.0:1787695655.051084:0:99012:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x56b:0x0](ffff9d85b3cff448) 00000080:00200000:0.0:1787695655.059622:0:99013:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695655.059650:0:99013:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.059721:0:99013:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99013 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.059731:0:99013:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99013 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.059743:0:99013:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99013 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695655.059772:0:99013:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99013:1874531134095744:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:0.0:1787695655.059915:0:99013:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:0.0:1787695655.065223:0:99013:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99013:1874531134095744:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695655.065256:0:99013:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065270:0:99013:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065283:0:99013:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065293:0:99013:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065305:0:99013:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065313:0:99013:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065319:0:99013:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695655.065333:0:99013:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d0a680 x1874531134095744/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695655.065352:0:99013:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065365:0:99013:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695655.065372:0:99013:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695655.065414:0:99013:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695655.065420:0:99013:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293dd 00000080:00010000:0.0:1787695655.065426:0:99013:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695655.065429:0:99013:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293dd from it ffffb5d404343be0 00010000:00010000:0.0:1787695655.065433:0:99013:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.065442:0:99013:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695655.065468:0:99013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695655.065490:0:99013:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695655.065495:0:99013:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695655.065508:0:99013:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.065517:0:99013:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.065527:0:99013:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695655.065534:0:99013:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695655.065538:0:99013:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695655.065541:0:99013:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293dd 00000080:00010000:0.0:1787695655.065545:0:99013:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293dd from it ffffb5d404343ce0 00010000:00010000:0.0:1787695655.065549:0:99013:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.065556:0:99013:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695655.065565:0:99013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695655.065570:0:99013:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695655.065576:0:99013:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.065582:0:99013:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695655.065590:0:99013:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695655.065597:0:99013:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695655.065600:0:99013:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695655.065603:0:99013:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293dd 00000080:00010000:0.0:1787695655.065606:0:99013:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293dd from it ffffb5d404343dc8 00010000:00010000:0.0:1787695655.065610:0:99013:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695655.065617:0:99013:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695655.067268:0:99013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695655.067285:0:99013:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695655.067337:0:99013:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695655.067415:0:99013:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695655.067424:0:99013:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695655.067427:0:99013:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898d800/0x678ae32e953293dd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef913 expref: -99 pid: 99013 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695655.067458:0:99013:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695655.067471:0:99013:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d0a680 x1874531134096000/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695655.067496:0:99013:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99013:1874531134096000:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695655.067587:0:99013:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:0.0:1787695655.079253:0:99013:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99013:1874531134096000:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695655.079315:0:99013:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x56c:0x0](ffff9d85b3cd42c8) 00000080:00200000:0.0:1787695655.079322:0:99013:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x56c:0x0], updating i_size 512 00000080:00200000:0.0:1787695655.085369:0:99014:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695655.085417:0:99014:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.085434:0:99014:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695655.085508:0:99014:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 99014 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.085519:0:99014:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99014 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.085534:0:99014:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99014 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695655.085563:0:99014:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99014:1874531134096128:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695655.085653:0:99014:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:0.0:1787695655.096687:0:99014:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99014:1874531134096128:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695655.096715:0:99014:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096726:0:99014:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096739:0:99014:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096750:0:99014:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096764:0:99014:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096771:0:99014:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096779:0:99014:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695655.096795:0:99014:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d08380 x1874531134096128/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695655.096810:0:99014:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096823:0:99014:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695655.096830:0:99014:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695655.096899:0:99014:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695655.096907:0:99014:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953293e4 00000080:00010000:0.0:1787695655.096922:0:99014:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695655.096925:0:99014:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293e4 from it ffffb5d40434bc20 00010000:00010000:0.0:1787695655.096929:0:99014:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.096938:0:99014:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695655.096964:0:99014:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:0.0:1787695655.096988:0:99014:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:0.0:1787695655.096993:0:99014:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:0.0:1787695655.097014:0:99014:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787695655.097055:0:99014:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 99014 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.097063:0:99014:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99014 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.097077:0:99014:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99014 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695655.097096:0:99014:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99014:1874531134096256:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695655.097158:0:99014:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:0.0:1787695655.108831:0:99014:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99014:1874531134096256:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:0.0:1787695655.108926:0:99014:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.108953:0:99014:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.108963:0:99014:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.108977:0:99014:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.108991:0:99014:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.109006:0:99014:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.109015:0:99014:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.109030:0:99014:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695655.109052:0:99014:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d08380 x1874531134096256/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 584/584 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695655.109066:0:99014:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.109077:0:99014:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935617/1388 (0 0) 00000080:00200000:0.0:1787695655.109084:0:99014:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695655.109122:0:99014:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x56c:0x0](ffff9d85b3cd6c08) 00000080:00200000:0.0:1787695655.109131:0:99014:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x56c:0x0], updating i_size 512 00000080:00010000:0.0:1787695655.109138:0:99014:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x56c:0x0](ffff9d85b3cd6c08) for lock 0x678ae32e953293eb 00000080:00010000:0.0:1787695655.109150:0:99014:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x56c:0x0](ffff9d85b3cd6c08) 00000080:00010000:0.0:1787695655.109154:0:99014:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293eb from it ffffb5d40434be08 00010000:00010000:0.0:1787695655.109157:0:99014:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.109166:0:99014:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695655.109199:0:99014:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695655.109217:0:99014:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.109228:0:99014:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.109244:0:99014:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695655.109250:0:99014:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695655.109255:0:99014:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:0.0:1787695655.109258:0:99014:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953293e4 00000080:00010000:0.0:1787695655.109262:0:99014:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293e4 from it ffffb5d40434bd90 00010000:00010000:0.0:1787695655.109265:0:99014:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.109272:0:99014:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695655.109280:0:99014:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:0.0:1787695655.109287:0:99014:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88) 00010000:00010000:0.0:1787695655.109310:0:99014:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695655.109345:0:99014:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.109394:0:99014:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 1/0,0 mode: --/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.109402:0:99014:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695655.109413:0:99014:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787695655.109416:0:99014:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898c800/0x678ae32e953293eb lrc: 0/0,0 mode: --/PR res: [0x200000401:0x56c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef94b expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.109438:0:99014:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293e4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef92f expref: -99 pid: 99014 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1787695655.109456:0:99014:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695655.109463:0:99014:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d08380 x1874531134096512/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695655.109490:0:99014:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99014:1874531134096512:192.168.201.137@tcp:36:rmdir.0 00000100:00100000:0.0:1787695655.109566:0:99014:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:0.0:1787695655.114939:0:99014:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99014:1874531134096512:192.168.201.137@tcp:36:rmdir.0 00000080:00200000:0.0:1787695655.114992:0:99014:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x56c:0x0](ffff9d85b3cd6c08) 00000100:00100000:0.0:1787695655.141107:0:99015:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99015:1874531134096640:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:0.0:1787695655.141141:0:99015:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141154:0:99015:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141166:0:99015:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141176:0:99015:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141191:0:99015:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141199:0:99015:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141206:0:99015:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695655.141218:0:99015:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce1c00 x1874531134096640/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695655.141230:0:99015:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141243:0:99015:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695655.141249:0:99015:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695655.141279:0:99015:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695655.141285:0:99015:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293f2 00000080:00010000:0.0:1787695655.141292:0:99015:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695655.141295:0:99015:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293f2 from it ffffb5d404353be0 00010000:00010000:0.0:1787695655.141299:0:99015:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.141308:0:99015:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695655.141339:0:99015:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695655.141366:0:99015:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:0.0:1787695655.141371:0:99015:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695655.141396:0:99015:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.141406:0:99015:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.141419:0:99015:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695655.141426:0:99015:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695655.141430:0:99015:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695655.141433:0:99015:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293f2 00000080:00010000:0.0:1787695655.141437:0:99015:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293f2 from it ffffb5d404353ce0 00010000:00010000:0.0:1787695655.141440:0:99015:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695655.141447:0:99015:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695655.141455:0:99015:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695655.141461:0:99015:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695655.141466:0:99015:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.141472:0:99015:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695655.141484:0:99015:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695655.141491:0:99015:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695655.141494:0:99015:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695655.141496:0:99015:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953293f2 00000080:00010000:0.0:1787695655.141499:0:99015:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293f2 from it ffffb5d404353dc8 00010000:00010000:0.0:1787695655.141502:0:99015:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695655.141511:0:99015:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695655.141518:0:99015:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695655.141524:0:99015:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695655.141559:0:99015:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695655.141618:0:99015:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695655.141637:0:99015:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695655.141640:0:99015:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e953293f2 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef96e expref: -99 pid: 99015 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695655.141667:0:99015:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695655.141675:0:99015:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d08a80 x1874531134096896/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695655.141699:0:99015:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99015:1874531134096896:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:0.0:1787695655.141783:0:99015:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:0.0:1787695655.145759:0:99015:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99015:1874531134096896:192.168.201.137@tcp:36:mkdir.0 00000080:00200000:0.0:1787695655.145825:0:99015:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x56d:0x0](ffff9d85b3cd6c08) 00000080:00200000:0.0:1787695655.145840:0:99015:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000401:0x56d:0x0], updating i_size 512 00000080:00200000:0.0:1787695655.156045:0:99016:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695655.156076:0:99016:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695655.156092:0:99016:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787695655.156151:0:99016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 99016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.156162:0:99016:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695655.156172:0:99016:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 99016 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695655.156201:0:99016:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99016:1874531134097024:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:0.0:1787695655.156283:0:99016:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:2.0:1787695655.167043:0:99016:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09880 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99016:1874531134097024:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695655.167073:0:99016:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167086:0:99016:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167097:0:99016:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167108:0:99016:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167120:0:99016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167129:0:99016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167136:0:99016:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695655.167153:0:99016:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d09880 x1874531134097024/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695655.167167:0:99016:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167180:0:99016:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695655.167187:0:99016:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695655.167218:0:99016:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695655.167224:0:99016:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953293f9 00000080:00010000:2.0:1787695655.167229:0:99016:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695655.167231:0:99016:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953293f9 from it ffffb5d40425bc20 00010000:00010000:2.0:1787695655.167235:0:99016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167243:0:99016:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d800/0x678ae32e953293f9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef98a expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695655.167269:0:99016:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695655.167292:0:99016:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), flags=0 00000080:00200000:2.0:1787695655.167296:0:99016:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=getattr 00000002:00010000:2.0:1787695655.167314:0:99016:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d43a.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695655.167352:0:99016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329400 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 99016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167360:0:99016:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329400 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.167373:0:99016:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329400 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 99016 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695655.167404:0:99016:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99016:1874531134097152:192.168.201.137@tcp:101:rmdir.0 00000100:00100000:2.0:1787695655.167497:0:99016:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695655.170652:0:99016:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job rmdir:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99016:1874531134097152:192.168.201.137@tcp:101:rmdir.0 00010000:00010000:2.0:1787695655.170683:0:99016:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86e400/0x678ae32e95329400 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9a6 expref: -99 pid: 99016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787695655.177216:0:99016:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x56d:0x0](ffff9d85b3d02a08) 00000080:00200000:2.0:1787695655.186723:0:99017:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695655.186754:0:99017:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695655.186823:0:99017:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99017 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.186833:0:99017:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99017 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.186844:0:99017:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99017 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695655.186917:0:99017:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99017:1874531134097536:192.168.201.137@tcp:101:mkdir.0 00000100:00100000:2.0:1787695655.187007:0:99017:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00000100:00100000:2.0:1787695655.190827:0:99017:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99017:1874531134097536:192.168.201.137@tcp:101:mkdir.0 00010000:00010000:2.0:1787695655.190893:0:99017:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191052:0:99017:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191064:0:99017:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191076:0:99017:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191090:0:99017:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191099:0:99017:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191107:0:99017:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695655.191119:0:99017:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd7100 x1874531134097536/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695672 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695655.191134:0:99017:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191149:0:99017:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695655.191156:0:99017:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695655.191187:0:99017:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695655.191194:0:99017:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329407 00000080:00010000:2.0:1787695655.191200:0:99017:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695655.191204:0:99017:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329407 from it ffffb5d404313be0 00010000:00010000:2.0:1787695655.191208:0:99017:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695655.191217:0:99017:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695655.191244:0:99017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695655.191269:0:99017:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=1536 00000080:00200000:2.0:1787695655.191275:0:99017:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695655.191290:0:99017:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695655.191300:0:99017:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695655.191311:0:99017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695655.191319:0:99017:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695655.191323:0:99017:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695655.191326:0:99017:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329407 00000080:00010000:2.0:1787695655.191330:0:99017:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329407 from it ffffb5d404313ce0 00010000:00010000:2.0:1787695655.191334:0:99017:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695655.191342:0:99017:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695655.191358:0:99017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695655.191364:0:99017:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695655.191370:0:99017:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695655.191376:0:99017:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695655.191397:0:99017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695655.191403:0:99017:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695655.191407:0:99017:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695655.191410:0:99017:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329407 00000080:00010000:2.0:1787695655.191414:0:99017:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329407 from it ffffb5d404313dc8 00010000:00010000:2.0:1787695655.191417:0:99017:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695655.191424:0:99017:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695655.191432:0:99017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695655.191438:0:99017:0:(namei.c:2239:do_mkdir()) VFS Op:name=d43a.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695655.191475:0:99017:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695655.191527:0:99017:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695655.191542:0:99017:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695655.191545:0:99017:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e95329407 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832ef9c9 expref: -99 pid: 99017 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695655.191573:0:99017:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695655.191581:0:99017:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd7100 x1874531134097792/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695655.191605:0:99017:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job mkdir:68e474e8-42cb-4207-86a3-041065654492:99017:1874531134097792:192.168.201.137@tcp:36:mkdir.0 00000100:00100000:2.0:1787695655.191886:0:99017:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000600 going to sleep for 12 seconds 00010000:00010000:0.0:1787695656.147546:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695658.195203:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695660.243432:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695661.282284:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695662.291319:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695663.316049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695663.797629:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134099456:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695663.797660:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134099072:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695664.341832:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1787695664.996753:0:99575:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanityn test 43b: pdirops: unlink vs create ======================================================================== 18:07:43 (1787695663) 00010000:00010000:2.0:1787695665.095305:0:99580:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9d8590d43000 00010000:00010000:0.0:1787695665.096905:0:6059:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 86 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787695665.096920:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e953241f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d964a expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.096956:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x4:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.096994:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e953241ea lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x24:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d962e expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.097270:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86fe00/0x678ae32e953241e3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x24:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9620 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.097296:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86fa00/0x678ae32e953241dc lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x186:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9604 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.097312:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8580031e00/0x678ae32e953241c7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d95d3 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097322:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x66:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097336:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d6600/0x678ae32e953241c0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x2d:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d95b7 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.097353:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d7000/0x678ae32e953241b9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x2d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d95a9 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.097368:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f800/0x678ae32e953241ab lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9586 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097376:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x4e:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097388:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f200/0x678ae32e9532419d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9563 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097397:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x5a:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097408:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ea00/0x678ae32e9532418f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9540 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097417:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x60:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097428:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c2200/0x678ae32e95324181 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d951d expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097438:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x89:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097449:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c2a00/0x678ae32e9532417a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x91:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9501 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.097466:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c2800/0x678ae32e95324173 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x91:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d94f3 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.097482:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c2400/0x678ae32e95324165 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d94d0 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097492:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x7c:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097505:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c3000/0x678ae32e9532415e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x27:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d94b4 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.097519:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c3400/0x678ae32e95324157 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x27:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d94a6 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.097536:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c3e00/0x678ae32e95324149 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9483 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097546:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0xa1:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097558:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d7800/0x678ae32e9532413b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9460 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097572:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x77:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097595:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c3600/0x678ae32e9532412d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d943d expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097607:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x34:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097619:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d7600/0x678ae32e9532411f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d941a expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097628:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x6:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097639:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c2600/0x678ae32e9532410a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d93f0 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097648:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x38:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097668:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85852c2000/0x678ae32e953240fc lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d93cd expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097679:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x7e:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097690:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f00c00/0x678ae32e953240ee lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d93aa expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097707:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x64:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.097720:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f00800/0x678ae32e953240e0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9387 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.097729:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x5:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.100240:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f00600/0x678ae32e953240d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9e:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d936b expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.100287:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f01200/0x678ae32e953240d2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d935d expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.100328:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f00a00/0x678ae32e953240c4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d933a expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.100344:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0xa2:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.100361:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f00e00/0x678ae32e953240bd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x83:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d931e expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.100380:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f00400/0x678ae32e953240b6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x83:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9310 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.102420:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f01e00/0x678ae32e953240a8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d92ed expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.102454:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x9f:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.102478:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f00200/0x678ae32e9532409a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d92ca expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.102488:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.102498:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d8590f01c00/0x678ae32e9532408c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d92a7 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.102508:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x59:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.102518:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679d400/0x678ae32e9532407e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9284 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.102532:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x62:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.102542:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679de00/0x678ae32e95324070 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9261 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.102551:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x5c:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.102560:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679da00/0x678ae32e95324069 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x21:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9245 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.102579:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679d600/0x678ae32e95324062 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x21:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9237 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.103111:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679d800/0x678ae32e95324054 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9214 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103127:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x5f:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103138:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679d200/0x678ae32e95324046 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d91f1 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103147:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x61:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103156:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679ca00/0x678ae32e95324038 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d91ce expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103164:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x78:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103174:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858679ce00/0x678ae32e9532402a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d91ab expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103182:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x3b:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103191:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09e600/0x678ae32e95324015 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9181 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103201:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x13:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103222:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09f200/0x678ae32e9532400e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x90:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9165 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.103239:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d6000/0x678ae32e95324007 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x90:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9157 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.103256:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09ea00/0x678ae32e95323ff9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9134 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103266:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x35:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103277:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09e800/0x678ae32e95323ff2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1c:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9118 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.103292:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d7200/0x678ae32e95323feb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d910a expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.103306:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d6800/0x678ae32e95323fdd lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d90e7 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103315:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x23:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103325:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d7a00/0x678ae32e95323fcf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d90c4 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103333:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x1b:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103342:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d6400/0x678ae32e95323fc8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x27:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d90a8 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.103363:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d6e00/0x678ae32e95323fc1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x27:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d909a expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.103377:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d6c00/0x678ae32e95323fb3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9077 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103386:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x5d:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103397:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b72d6a00/0x678ae32e95323fa5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9054 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103406:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x86:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103415:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09f400/0x678ae32e95323f97 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d9031 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103424:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x3a:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103434:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09ec00/0x678ae32e95323f90 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x2b:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9015 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.103449:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936ee00/0x678ae32e95323f89 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x2b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d9007 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.103464:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e200/0x678ae32e95323f82 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x47:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d8feb expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.103481:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e800/0x678ae32e95323f7b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x47:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d8fdd expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.103495:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f400/0x678ae32e95323f74 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x73:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d8fc1 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.103508:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936e400/0x678ae32e95323f6d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x73:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d8fb3 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695665.103522:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898d000/0x678ae32e95323f5f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d8f90 expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103530:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x63:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103542:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09fa00/0x678ae32e95323f51 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d8f6d expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103551:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x28:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103563:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858a09f600/0x678ae32e95323f4a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d8f51 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103572:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x7b:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103583:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b9d75e00/0x678ae32e95323f3c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d8f2e expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103603:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x33:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103614:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858b96da00/0x678ae32e95323f2e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832d8f0b expref: -99 pid: 96400 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695665.103622:0:6059:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x2:0x0]: losing layout lock 00010000:00010000:0.0:1787695665.103641:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858b96d600/0x678ae32e95323f27 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x29:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d8eef expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695665.103656:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85b9d74e00/0x678ae32e95323f20 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x29:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832d8ee1 expref: -99 pid: 96400 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695666.387101:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695667.411165:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787695668.185296:0:99690:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695668.185330:0:99690:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695668.185396:0:99690:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.185408:0:99690:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.185418:0:99690:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695668.185444:0:99690:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134099840:192.168.201.137@tcp:101:touch.0 00000100:00100000:0.0:1787695668.185559:0:99690:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695668.193410:0:99690:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134099840:192.168.201.137@tcp:101:touch.0 00010000:00010000:0.0:1787695668.193436:0:99690:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193449:0:99690:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193460:0:99690:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193470:0:99690:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193488:0:99690:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193495:0:99690:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193502:0:99690:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695668.193513:0:99690:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d09500 x1874531134099840/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695685 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695668.193527:0:99690:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193540:0:99690:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695668.193546:0:99690:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695668.193572:0:99690:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695668.193578:0:99690:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532941c 00000080:00010000:0.0:1787695668.193583:0:99690:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695668.193587:0:99690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532941c from it ffffb5d40190bbf8 00010000:00010000:0.0:1787695668.193591:0:99690:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193598:0:99690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695668.193626:0:99690:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695668.193646:0:99690:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695668.193658:0:99690:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695668.193667:0:99690:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695668.193677:0:99690:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695668.193682:0:99690:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695668.193686:0:99690:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695668.193689:0:99690:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532941c 00000080:00010000:0.0:1787695668.193692:0:99690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532941c from it ffffb5d40190bb38 00010000:00010000:0.0:1787695668.193695:0:99690:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695668.193702:0:99690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695668.193709:0:99690:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695668.193714:0:99690:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f43b.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), file ffff9d85c0fe2000, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1787695668.193721:0:99690:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f43b.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000002:00010000:0.0:1787695668.193740:0:99690:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f43b.sanityn,[0x200000401:0x56f:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1787695668.193750:0:99690:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695668.193795:0:99690:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695668.193802:0:99690:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695668.193805:0:99690:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c600/0x678ae32e9532941c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efa24 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695668.193837:0:99690:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695668.193844:0:99690:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787695668.193874:0:99690:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787695668.193881:0:99690:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d09500 x1874531134100096/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787695668.193891:0:99690:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695668.193911:0:99690:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100096:192.168.201.137@tcp:101:touch.0 00000100:00100000:0.0:1787695668.194030:0:99690:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c13c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695668.198914:0:99690:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100096:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695668.199125:0:99690:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199163:0:99690:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 4/0,1 mode: --/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199179:0:99690:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 4/0,1 mode: --/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199195:0:99690:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199207:0:99690:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199221:0:99690:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199229:0:99690:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199459:0:99690:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695668.199474:0:99690:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9d85b9d09500 x1874531134100096/t4294997959(4294997959) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695685 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787695668.199486:0:99690:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85b9d09500 x1874531134100096/t4294997959(4294997959) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695685 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695668.199500:0:99690:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695668.199511:0:99690:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.199521:0:99690:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589935617/1391 (0 0) 00000080:00200000:2.0:1787695668.199528:0:99690:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695668.199540:0:99690:0:(namei.c:1628:ll_create_it()) VFS Op:name=f43b.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000080:00200000:2.0:1787695668.199568:0:99690:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x56f:0x0](ffff9d85b3cff448) 00000080:00200000:2.0:1787695668.199830:0:99690:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x56f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1787695668.199836:0:99690:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000401:0x56f:0x0](ffff9d85b3cff448) 00000080:00010000:2.0:1787695668.199840:0:99690:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x56f:0x0](ffff9d85b3cff448) for lock 0x678ae32e95329423 00000080:00010000:2.0:1787695668.199883:0:99690:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x56f:0x0](ffff9d85b3cff448) 00000080:00200008:2.0:1787695668.199899:0:99690:0:(file.c:1090:ll_file_open()) START file f43b.sanityn:[0x200000401:0x56f:0x0](ffff9d85b3cff448), flags 104501 00000002:00100000:2.0:1787695668.199918:0:99690:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85b9d09500 x1874531134100096/t4294997959(4294997959) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695685 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787695668.200403:0:99690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329423 from it ffff9d85990859c0 00010000:00010000:2.0:1787695668.200410:0:99690:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.200420:0:99690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695668.200442:0:99690:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000401:0x56f:0x0](ffff9d85b3cff448), inode mode 81a4 mask 40 00000080:00200008:2.0:1787695668.200481:0:99690:0:(llite_lib.c:2427:ll_setattr_raw()) START file f43b.sanityn:[0x200000401:0x56f:0x0](ffff9d85b3cff448) current size 0, valid attrs 20070, mode 9d85, uid 26263120, gid -18988, new size 18446744072543992343, atime 1787695668.000000000, mtime 1787695668.000000000, ctime 1787695668.000000000 00010000:00010000:2.0:1787695668.200501:0:99690:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695668.200517:0:99690:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x56f:0x0]: losing layout lock 00010000:00010000:2.0:1787695668.200553:0:99690:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695668.200560:0:99690:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695668.200563:0:99690:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898cc00/0x678ae32e95329423 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efa40 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787695668.209942:0:99690:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695668.209963:0:99690:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd6680 x1874531134100352/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695668.210006:0:99690:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100352:192.168.201.137@tcp:36:touch.0 00000100:00100000:2.0:1787695668.210109:0:99690:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695668.221585:0:99690:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100352:192.168.201.137@tcp:36:touch.0 00000080:00200000:2.0:1787695668.221641:0:99690:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x56f:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1787695668.221664:0:99690:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000401:0x56f:0x0](ffff9d85b3cff448) 00000002:00010000:2.0:1787695668.221674:0:99690:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x56f:0x0]) in obj [0x200000401:0x56f:0x0], intent: layout flags 0 00010000:00010000:2.0:1787695668.221728:0:99690:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.221739:0:99690:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.221749:0:99690:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99690 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695668.221770:0:99690:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100480:192.168.201.137@tcp:101:touch.0 00000100:00100000:2.0:1787695668.221841:0:99690:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8583000840 going to sleep for 12 seconds 00000100:00100000:2.0:1787695668.226703:0:99690:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100480:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695668.226743:0:99690:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.226766:0:99690:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695668.226780:0:99690:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9d85babd5500 x1874531134100480/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 376/400 e 0 to 0 dl 1787695685 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695668.226791:0:99690:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695668.226805:0:99690:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.226819:0:99690:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 16 res 8589935617/1391 (0 0) 00000080:00200000:2.0:1787695668.226825:0:99690:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000401:0x56f:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787695668.226845:0:99690:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x56f:0x0](ffff9d85b3cff448) for lock 0x678ae32e9532942a 00000080:00010000:2.0:1787695668.227673:0:99690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532942a from it ffffb5d40190ba78 00010000:00010000:2.0:1787695668.227681:0:99690:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695668.227693:0:99690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695668.227720:0:99690:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695668.227731:0:99690:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787695668.227744:0:99690:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000401:0x56f:0x0](00000000428aafb8) being reconfigured ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695668.227795:0:99690:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x56f:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1787695668.227800:0:99690:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695668.227808:0:99690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787695668.227830:0:99690:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00080000:2.0:1787695668.227894:0:99690:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3413 00000100:00080000:2.0:1787695668.227906:0:99690:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695668.227916:0:99690:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 3407872, lost: 0. 00000100:00100000:2.0:1787695668.228550:0:99690:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134100608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695668.228648:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:2.0:1787695668.228665:0:2362:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d85babd5500 x1874531134100736/t0(0) o2->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787695668.228888:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134100608:192.168.201.137@tcp:8:touch.0 00000100:00100000:2.0:1787695668.231527:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d8590d43000: committing for last_committed 4294969643 gen 16 00000100:00080000:2.0:1787695668.231544:0:2359:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff9d8590d43000: connect to target with instance 1 00000100:00080000:2.0:1787695668.231555:0:2359:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff9d8590d43000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1787695668.231568:0:2359:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1787695668.231572:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1787695668.231594:0:2359:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1787695668.231600:0:2359:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1787695668.231615:0:2359:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000c49ee780): req@ffff9d85babd5500 x1874531134100736/t0(0) o2->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787695668.231643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134100608:192.168.201.137@tcp:8:touch.0 00000080:00200000:2.0:1787695668.294792:0:99690:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x56f:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787695668.294821:0:99690:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f43b.sanityn:[0x200000401:0x56f:0x0](ffff9d85b3cff448) current size 0, valid attrs 100201f0, mode 9d85, uid 26263120, gid -18988, new size 18446744072543992343, atime 1787695668.000000000, mtime 1787695668.000000000, ctime 1787695668.000000000, rc 0 00000080:00200008:2.0:1787695668.294880:0:99690:0:(file.c:479:ll_file_release()) START file f43b.sanityn:[0x200000401:0x56f:0x0](ffff9d85b3cff448), flags 104001 00000002:00100000:0.0:1787695668.295819:0:99690:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d85b9d09500 x1874531134100096/t4294997959(4294997959) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695685 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787695668.295833:0:99690:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787695668.295838:0:99690:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd7b80 x1874531134100992/t0(0) o35->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787695668.295880:0:99690:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100992:192.168.201.137@tcp:35:touch.0 00000100:00100000:0.0:1787695668.295975:0:99690:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1780 going to sleep for 11 seconds 00000100:00100000:2.0:1787695668.320449:0:99690:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:99690:1874531134100992:192.168.201.137@tcp:35:touch.0 00000002:00100000:2.0:1787695668.320481:0:99690:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d85b9d09500 x1874531134100096/t4294997959(4294997959) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695685 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787695668.884040:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134101248:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695668.884416:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134101504:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695668.898204:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134101248:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695668.898227:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134101504:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695669.459101:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787695669.671265:0:99751:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695669.671299:0:99751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695669.671379:0:99751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695669.671393:0:99751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695669.671404:0:99751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695669.671434:0:99751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:99751:1874531134101632:192.168.201.137@tcp:101:rm.0 00000100:00100000:0.0:1787695669.671663:0:99751:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85829c1180 going to sleep for 12 seconds 00000100:00100000:0.0:1787695669.689041:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:68e474e8-42cb-4207-86a3-041065654492:2361:1874531134101888:192.168.201.137@tcp:101:rm.0 00000080:00200000:0.0:1787695669.693240:0:99751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x56f:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1787695669.693247:0:99751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8588a56750 (ffff9d85a93ebea8/0) at cl_lock_release():245 00000020:00010000:0.0:1787695669.693250:0:99751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8588a56750 (ffff9d85a93ebea8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787695669.693255:0:99751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d85c1e9e330 (ffff9d85a93eb1b8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787695669.693298:0:99751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8588a56750 (ffff9d85a93ebea8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787695669.693301:0:99751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d85c1e9e330 (ffff9d85a93ebea8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787695669.693337:0:99751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x56f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787695669.693410:0:99751:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695669.693435:0:99751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695669.693450:0:99751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695669.693462:0:99751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695669.693469:0:99751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695669.693476:0:99751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695669.693480:0:99751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329431 00000080:00010000:0.0:1787695669.693486:0:99751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329431 from it ffffb5d401f1fc10 00010000:00010000:0.0:1787695669.693489:0:99751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695669.693497:0:99751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695669.693507:0:99751:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695669.693518:0:99751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f43b.sanityn, flags=0 00000080:00200000:0.0:1787695669.693523:0:99751:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695669.693528:0:99751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695669.693534:0:99751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695669.693547:0:99751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695669.693553:0:99751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695669.693556:0:99751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695669.693559:0:99751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329431 00000080:00010000:0.0:1787695669.693563:0:99751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329431 from it ffffb5d401f1fd68 00010000:00010000:0.0:1787695669.693566:0:99751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695669.693573:0:99751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695669.693579:0:99751:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695669.693659:0:99751:0:(namei.c:2418:ll_unlink()) VFS Op:name=f43b.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695669.693693:0:99751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695669.693721:0:99751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e95329438 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efa86 expref: -99 pid: 99751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695669.693731:0:99751:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x56f:0x0]: losing layout lock 00010000:00010000:0.0:1787695669.693742:0:99751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695669.693748:0:99751:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x56f:0x0]: losing layout lock 00010000:00010000:0.0:1787695669.693777:0:99751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e95329438 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efa86 expref: -99 pid: 99751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695669.693784:0:99751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 1/0,0 mode: --/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695669.693791:0:99751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695669.693796:0:99751:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1787695669.693799:0:99751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e95329438 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x56f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efa86 expref: -99 pid: 99751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695669.693814:0:99751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ea00/0x678ae32e9532942a lrc: 0/0,0 mode: --/CR res: [0x200000401:0x56f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efa55 expref: -99 pid: 99690 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695669.693826:0:99751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898dc00/0x678ae32e95329431 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efa6a expref: -99 pid: 99751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695669.693842:0:99751:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695669.694083:0:99751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d09f80 x1874531134102144/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000080:00200000:2.0:1787695670.197073:0:99753:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695670.197111:0:99753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695670.201588:0:99753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695670.201606:0:99753:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695670.201623:0:99753:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99753 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695670.201655:0:99753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job multiop:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99753:1874531134102272:192.168.201.137@tcp:101:multiop.0 00000100:00100000:2.0:1787695670.201770:0:99753:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85c1eb5480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695672.068322:0:99753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job multiop:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99753:1874531134102272:192.168.201.137@tcp:101:multiop.0 00010000:00010000:2.0:1787695672.068346:0:99753:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068358:0:99753:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068370:0:99753:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068381:0:99753:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068393:0:99753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068402:0:99753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068409:0:99753:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695672.068420:0:99753:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd6a00 x1874531134102272/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695687 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695672.068433:0:99753:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068444:0:99753:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695672.068450:0:99753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695672.068474:0:99753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695672.068480:0:99753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329446 00000080:00010000:2.0:1787695672.068485:0:99753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695672.068488:0:99753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329446 from it ffffb5d401f37bf8 00010000:00010000:2.0:1787695672.068491:0:99753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068498:0:99753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695672.068522:0:99753:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 1 00000080:00200000:2.0:1787695672.068542:0:99753:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:2.0:1787695672.068566:0:99753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695672.068575:0:99753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.068586:0:99753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695672.068592:0:99753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695672.068597:0:99753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695672.068600:0:99753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e95329446 00000080:00010000:2.0:1787695672.068603:0:99753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329446 from it ffffb5d401f37b38 00010000:00010000:2.0:1787695672.068606:0:99753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.068614:0:99753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695672.068637:0:99753:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695672.068642:0:99753:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f43b.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), file ffff9d85875bf680, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:2.0:1787695672.068650:0:99753:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f43b.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=open|creat 00000002:00010000:2.0:1787695672.068669:0:99753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f43b.sanityn,[0x200000402:0x37bf:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:2.0:1787695672.068681:0:99753:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695672.068721:0:99753:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695672.068728:0:99753:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695672.068731:0:99753:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86f400/0x678ae32e95329446 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efaa9 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695672.068761:0:99753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.068768:0:99753:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99753 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787695672.068778:0:99753:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787695672.068784:0:99753:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd6a00 x1874531134102528/t0(0) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787695672.068793:0:99753:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99753 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695672.068811:0:99753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job multiop:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99753:1874531134102528:192.168.201.137@tcp:101:multiop.0 00000100:00100000:2.0:1787695672.068986:0:99753:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85c1eb5480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695672.082061:0:99753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job multiop:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99753:1874531134102528:192.168.201.137@tcp:101:multiop.0 00010000:00010000:2.0:1787695672.082102:0:99753:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082134:0:99753:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 4/0,1 mode: --/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082142:0:99753:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 4/0,1 mode: --/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082152:0:99753:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082163:0:99753:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082176:0:99753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082183:0:99753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082190:0:99753:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695672.082215:0:99753:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9d85babd6a00 x1874531134102528/t4294997963(4294997963) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 656/608 e 0 to 0 dl 1787695689 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787695672.082229:0:99753:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85babd6a00 x1874531134102528/t4294997963(4294997963) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 656/608 e 0 to 0 dl 1787695689 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695672.082241:0:99753:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082251:0:99753:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d858a6f7400 type 13 mode 4 res 8589935618/14271 (0 0) 00000080:00200000:2.0:1787695672.082257:0:99753:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695672.082271:0:99753:0:(namei.c:1628:ll_create_it()) VFS Op:name=f43b.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), intent=open|creat 00000080:00200000:2.0:1787695672.082318:0:99753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x37bf:0x0](ffff9d85b3cff448) 00000080:00200000:2.0:1787695672.082325:0:99753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x37bf:0x0], updating i_size 0 00000080:00200000:2.0:1787695672.082363:0:99753:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x37bf:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1787695672.082368:0:99753:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x37bf:0x0](ffff9d85b3cff448) 00000080:00010000:2.0:1787695672.082371:0:99753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x37bf:0x0](ffff9d85b3cff448) for lock 0x678ae32e9532944d 00000080:00010000:2.0:1787695672.082381:0:99753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x37bf:0x0](ffff9d85b3cff448) 00000080:00200008:2.0:1787695672.082394:0:99753:0:(file.c:1090:ll_file_open()) START file f43b.sanityn:[0x200000402:0x37bf:0x0](ffff9d85b3cff448), flags 100300 00000002:00100000:2.0:1787695672.082412:0:99753:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85babd6a00 x1874531134102528/t4294997963(4294997963) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 656/608 e 0 to 0 dl 1787695689 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787695672.082448:0:99753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532944d from it ffff9d8587b87240 00010000:00010000:2.0:1787695672.082452:0:99753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.082462:0:99753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695672.082485:0:99753:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x37bf:0x0](ffff9d85b3cff448), inode mode 81a4 mask 40 00000080:00200008:2.0:1787695672.082517:0:99753:0:(file.c:479:ll_file_release()) START file f43b.sanityn:[0x200000402:0x37bf:0x0](ffff9d85b3cff448), flags 100000 00000002:00100000:2.0:1787695672.082560:0:99753:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d85babd6a00 x1874531134102528/t4294997963(4294997963) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 656/608 e 0 to 0 dl 1787695689 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787695672.082570:0:99753:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787695672.082575:0:99753:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd4700 x1874531134102784/t0(0) o35->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787695672.082594:0:99753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job multiop:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99753:1874531134102784:192.168.201.137@tcp:35:multiop.0 00000100:00100000:2.0:1787695672.082763:0:99753:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85c1eb5480 going to sleep for 11 seconds 00000100:00100000:2.0:1787695672.100264:0:99753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job multiop:b3d8e19c-573a-445a-9e98-5fe67f1b3715:99753:1874531134102784:192.168.201.137@tcp:35:multiop.0 00000002:00100000:2.0:1787695672.100297:0:99753:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d85babd6a00 x1874531134102528/t4294997963(4294997963) o101->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 656/608 e 0 to 0 dl 1787695689 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787695672.502554:0:2352:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.137@tcp (source: 12345-192.168.201.137@tcp) 00010000:00010000:2.0:1787695672.513651:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787695672.513662:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 000000009358677b/0x678ae32e9532943f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x81f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x297ceef0832efa8d expref: -99 pid: 99751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.513675:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8580030a00 already unused, calling callback (ffffffffc12f4300) 00010000:00010000:2.0:1787695672.513684:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 000000009358677b/0x678ae32e9532943f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x81f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x297ceef0832efa8d expref: -99 pid: 99751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.513738:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 000000009358677b/0x678ae32e9532943f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x81f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x297ceef0832efa8d expref: -99 pid: 99751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.513746:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695672.513772:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 000000009358677b/0x678ae32e9532943f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x81f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x297ceef0832efa8d expref: -99 pid: 99751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.513786:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 000000009358677b/0x678ae32e9532943f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x81f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x297ceef0832efa8d expref: -99 pid: 99751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.513810:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695672.513812:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695672.513814:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695672.513830:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134102912:192.168.201.137@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1787695672.654658:0:99445:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695672.654687:0:99445:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695672.654751:0:99445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 99445 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.654762:0:99445:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99445 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.654772:0:99445:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 99445 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695672.654798:0:99445:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job bash:68e474e8-42cb-4207-86a3-041065654492:99445:1874531134103040:192.168.201.137@tcp:101:bash.0 00000100:00100000:2.0:1787695672.655090:0:99445:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85c1eb5480 going to sleep for 12 seconds 00000100:00100000:2.0:1787695672.661026:0:99445:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d8590d43000: committing for last_committed 4294997962 gen 1 00000100:00100000:2.0:1787695672.661092:0:99445:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job bash:68e474e8-42cb-4207-86a3-041065654492:99445:1874531134103040:192.168.201.137@tcp:101:bash.0 00010000:00010000:2.0:1787695672.661124:0:99445:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661140:0:99445:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661154:0:99445:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661167:0:99445:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661180:0:99445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661189:0:99445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661196:0:99445:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695672.661208:0:99445:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd5c00 x1874531134103040/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695689 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695672.661222:0:99445:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661234:0:99445:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695672.661241:0:99445:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695672.661273:0:99445:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 11776 00000080:00010000:2.0:1787695672.661279:0:99445:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329454 00000080:00010000:2.0:1787695672.661285:0:99445:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695672.661289:0:99445:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329454 from it ffffb5d400d17bb0 00010000:00010000:2.0:1787695672.661293:0:99445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695672.661302:0:99445:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695672.661325:0:99445:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 44 00000080:00200008:2.0:1787695672.661339:0:99445:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags 304000 00000080:00200008:2.0:1787695672.661369:0:99445:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d858ad3ec08), request_mask 2047, flags 0, foreign 0 00000002:00010000:2.0:1787695672.661397:0:99445:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: readdir flags 0 00010000:00010000:2.0:1787695672.661408:0:99445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.661419:0:99445:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1787695672.662242:0:99445:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job bash:68e474e8-42cb-4207-86a3-041065654492:99445:1874531134103168:192.168.201.137@tcp:37:bash.0 00000100:00100000:2.0:1787695672.662415:0:99445:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85c1eb5480 going to sleep for 11 seconds 00000100:00100000:2.0:1787695672.680658:0:99445:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job bash:68e474e8-42cb-4207-86a3-041065654492:99445:1874531134103168:192.168.201.137@tcp:37:bash.0 00010000:00010000:2.0:1787695672.681366:0:99445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695672.681383:0:99445:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695672.681414:0:99445:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695672.681437:0:99445:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695672.681453:0:99445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695672.681465:0:99445:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695672.681473:0:99445:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695672.681479:0:99445:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695672.681483:0:99445:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329454 00000080:00010000:2.0:1787695672.681489:0:99445:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329454 from it ffffb5d400d17cd8 00010000:00010000:2.0:1787695672.681492:0:99445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695672.681499:0:99445:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f400/0x678ae32e95329454 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efaef expref: -99 pid: 99445 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695672.681707:0:99445:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08) pos/size0/11776 32bit_api 0 00000080:00200000:2.0:1787695672.681764:0:99445:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200008:2.0:1787695672.681779:0:99445:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags 304000 00000100:00100000:0.0:1787695672.713340:0:5588:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874531140562688 00000100:00100000:0.0:1787695672.713353:0:5588:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874531140562688 00000100:00100000:0.0:1787695672.713417:0:5588:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.137@tcp, seq: 10503 00000100:00100000:0.0:1787695672.713428:0:5588:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9d85b9d2d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:b3d8e19c-573a-445a-9e98-5fe67f1b3715+5:5845:x1874531140562688:12345-192.168.201.137@tcp:104: 00010000:00010000:0.0:1787695672.713446:0:5588:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d85ae86ec00/0x678ae32e9532944d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x37bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x297ceef0832efad3 expref: -99 pid: 99753 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695672.713544:0:5588:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787695672.713552:0:5588:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9d85b9d2d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:b3d8e19c-573a-445a-9e98-5fe67f1b3715+5:5845:x1874531140562688:12345-192.168.201.137@tcp:104: Request processed in 125us (338us total) trans 0 rc 0/0 00000100:00100000:0.0:1787695672.713559:0:5588:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.137@tcp, seq: 10503 00000100:00100000:2.0:1787695672.731516:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134103680:192.168.201.137@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787695673.557617:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695674.009273:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134103936:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695674.009387:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134104192:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695674.009469:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134104448:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695674.009503:0:2363:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1787695674.009513:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2363:1874531134103808:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695674.009540:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134104064:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695674.009563:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134104320:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695674.582111:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695677.656583:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787695679.136899:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2360:1874531134104576:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.137005:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134104832:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.137034:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134105088:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.137062:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:0.0:1787695679.137068:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134104704:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.137091:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134104960:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.137113:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134105216:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.140714:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d8590d43000: committing for last_committed 4294997965 gen 1 00000100:00100000:0.0:1787695679.140748:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134104704:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.140767:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134104960:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.140779:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134105216:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.140790:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2360:1874531134104576:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.140802:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134104832:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695679.140818:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134105088:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695679.699035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695683.795222:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787695684.180212:0:100467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100467:1874531134105344:192.168.201.137@tcp:101:touch.0 00000100:00100000:2.0:1787695684.184593:0:100467:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9eccd80 going to sleep for 12 seconds 00010000:00010000:2.0:1787695684.241604:0:100467:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 4/0,1 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.241624:0:100467:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 4/0,1 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.241637:0:100467:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.241651:0:100467:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.241671:0:100467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.241681:0:100467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.241688:0:100467:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695684.241708:0:100467:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9d85b9ce3480 x1874531134105600/t4294997966(4294997966) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695701 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787695684.241722:0:100467:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85b9ce3480 x1874531134105600/t4294997966(4294997966) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695701 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695684.241736:0:100467:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695684.241751:0:100467:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.241761:0:100467:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589935617/1392 (0 0) 00000080:00200000:2.0:1787695684.241768:0:100467:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695684.241781:0:100467:0:(namei.c:1628:ll_create_it()) VFS Op:name=f43c.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000080:00200000:2.0:1787695684.241839:0:100467:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x570:0x0](ffff9d85b3d0ec08) 00000080:00200000:2.0:1787695684.242038:0:100467:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x570:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1787695684.242044:0:100467:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) 00000080:00010000:2.0:1787695684.242047:0:100467:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) for lock 0x678ae32e95329469 00000080:00010000:2.0:1787695684.242058:0:100467:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) 00000080:00200008:2.0:1787695684.242073:0:100467:0:(file.c:1090:ll_file_open()) START file f43c.sanityn:[0x200000401:0x570:0x0](ffff9d85b3d0ec08), flags 104501 00000002:00100000:2.0:1787695684.242092:0:100467:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85b9ce3480 x1874531134105600/t4294997966(4294997966) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695701 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787695684.242126:0:100467:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329469 from it ffff9d858b82cb40 00010000:00010000:2.0:1787695684.242131:0:100467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.242147:0:100467:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695684.242169:0:100467:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000401:0x570:0x0](ffff9d85b3d0ec08), inode mode 81a4 mask 40 00000080:00200008:2.0:1787695684.242206:0:100467:0:(llite_lib.c:2427:ll_setattr_raw()) START file f43c.sanityn:[0x200000401:0x570:0x0](ffff9d85b3d0ec08) current size 0, valid attrs 20070, mode 9d85, uid 30867024, gid -18988, new size 18446744072543992343, atime 1787695684.000000000, mtime 1787695684.000000000, ctime 1787695684.000000000 00010000:00010000:2.0:1787695684.242229:0:100467:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695684.242245:0:100467:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x570:0x0]: losing layout lock 00010000:00010000:2.0:1787695684.242296:0:100467:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695684.242304:0:100467:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695684.242323:0:100467:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e600/0x678ae32e95329469 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efb51 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787695684.242346:0:100467:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695684.242353:0:100467:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd4000 x1874531134105856/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695684.242375:0:100467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100467:1874531134105856:192.168.201.137@tcp:36:touch.0 00000100:00100000:2.0:1787695684.242482:0:100467:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8bc00 going to sleep for 12 seconds 00000100:00100000:0.0:1787695684.243223:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2360:1874531134105984:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695684.243267:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134106240:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695684.243290:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134106496:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695684.243327:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1787695684.243333:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134106112:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695684.243357:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134106368:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695684.254019:0:100467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100467:1874531134105856:192.168.201.137@tcp:36:touch.0 00000080:00200000:2.0:1787695684.254068:0:100467:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x570:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1787695684.254091:0:100467:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000401:0x570:0x0](ffff9d85b3d0ec08) 00000002:00010000:2.0:1787695684.254102:0:100467:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x570:0x0]) in obj [0x200000401:0x570:0x0], intent: layout flags 0 00010000:00010000:2.0:1787695684.254150:0:100467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x570:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 100467 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.254159:0:100467:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100467 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.254169:0:100467:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100467 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695684.254187:0:100467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100467:1874531134106624:192.168.201.137@tcp:101:touch.0 00000100:00100000:2.0:1787695684.254257:0:100467:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8bc00 going to sleep for 12 seconds 00000100:00100000:2.0:1787695684.280432:0:100467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100467:1874531134106624:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695684.280465:0:100467:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.280489:0:100467:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695684.280504:0:100467:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9d85babd5f80 x1874531134106624/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 376/400 e 0 to 0 dl 1787695701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695684.280515:0:100467:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695684.280527:0:100467:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.280539:0:100467:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 16 res 8589935617/1392 (0 0) 00000080:00200000:2.0:1787695684.280546:0:100467:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000401:0x570:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787695684.280562:0:100467:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) for lock 0x678ae32e95329470 00000080:00010000:2.0:1787695684.280570:0:100467:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329470 from it ffffb5d401d6fa78 00010000:00010000:2.0:1787695684.280574:0:100467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.280584:0:100467:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695684.280615:0:100467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695684.280635:0:100467:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787695684.280643:0:100467:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000401:0x570:0x0](00000000e7ece8ff) being reconfigured ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695684.280678:0:100467:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x570:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1787695684.280684:0:100467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695684.280692:0:100467:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787695684.280711:0:100467:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00080000:2.0:1787695684.280745:0:100467:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3442 00000100:00080000:2.0:1787695684.280755:0:100467:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695684.280765:0:100467:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 3407872, lost: 0. 00000100:00100000:2.0:1787695684.280786:0:100467:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134106752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695684.281273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134106752:192.168.201.137@tcp:8:touch.0 00000100:00100000:0.0:1787695684.287021:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9d8590d43000: committing for last_committed 4294969721 gen 21 00000100:00080000:0.0:1787695684.287040:0:2359:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff9d8590d43000: connect to target with instance 1 00000100:00080000:0.0:1787695684.287050:0:2359:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff9d8590d43000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1787695684.287063:0:2359:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1787695684.287067:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1787695684.287077:0:2359:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1787695684.287082:0:2359:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1787695684.287096:0:2359:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000c49ee780): req@ffff9d85babd5f80 x1874531134106880/t0(0) o2->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787695684.287124:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134106752:192.168.201.137@tcp:8:touch.0 00000080:00200000:2.0:1787695684.396636:0:100467:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x570:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787695684.396653:0:100467:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f43c.sanityn:[0x200000401:0x570:0x0](ffff9d85b3d0ec08) current size 0, valid attrs 100201f0, mode 9d85, uid 30867024, gid -18988, new size 18446744072543992343, atime 1787695684.000000000, mtime 1787695684.000000000, ctime 1787695684.000000000, rc 0 00000080:00200008:2.0:1787695684.396687:0:100467:0:(file.c:479:ll_file_release()) START file f43c.sanityn:[0x200000401:0x570:0x0](ffff9d85b3d0ec08), flags 104001 00000002:00100000:2.0:1787695684.396747:0:100467:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d85b9ce3480 x1874531134105600/t4294997966(4294997966) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695701 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787695684.396759:0:100467:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787695684.396764:0:100467:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd7b80 x1874531134107136/t0(0) o35->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787695684.396783:0:100467:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100467:1874531134107136:192.168.201.137@tcp:35:touch.0 00000100:00100000:2.0:1787695684.396908:0:100467:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8bc00 going to sleep for 11 seconds 00000100:00100000:2.0:1787695684.411033:0:100467:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100467:1874531134107136:192.168.201.137@tcp:35:touch.0 00000002:00100000:2.0:1787695684.411076:0:100467:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d85b9ce3480 x1874531134105600/t4294997966(4294997966) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695701 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787695684.417413:0:100468:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695684.417441:0:100468:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695684.417515:0:100468:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.417526:0:100468:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.417542:0:100468:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695684.417569:0:100468:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107264:192.168.201.137@tcp:101:touch.0 00000100:00100000:2.0:1787695684.417659:0:100468:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8bc00 going to sleep for 12 seconds 00000100:00100000:2.0:1787695684.421714:0:100468:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107264:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695684.421742:0:100468:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421752:0:100468:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421763:0:100468:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421773:0:100468:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421784:0:100468:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421791:0:100468:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421797:0:100468:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695684.421813:0:100468:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd7480 x1874531134107264/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695684.421826:0:100468:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421837:0:100468:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695684.421843:0:100468:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695684.421953:0:100468:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695684.421960:0:100468:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329477 00000080:00010000:2.0:1787695684.421966:0:100468:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695684.421968:0:100468:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329477 from it ffffb5d401c97bf8 00010000:00010000:2.0:1787695684.421972:0:100468:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.421980:0:100468:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695684.422006:0:100468:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695684.422034:0:100468:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695684.422048:0:100468:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695684.422058:0:100468:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695684.422068:0:100468:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695684.422079:0:100468:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695684.422083:0:100468:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695684.422085:0:100468:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329477 00000080:00010000:2.0:1787695684.422089:0:100468:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329477 from it ffffb5d401c97b38 00010000:00010000:2.0:1787695684.422091:0:100468:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695684.422099:0:100468:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695684.422107:0:100468:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695684.422113:0:100468:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f43c.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), file ffff9d8590dac900, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1787695684.422122:0:100468:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f43c.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000002:00010000:2.0:1787695684.422211:0:100468:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f43c.sanityn-2,[0x200000401:0x571:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1787695684.422224:0:100468:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695684.422270:0:100468:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695684.422277:0:100468:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695684.422280:0:100468:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86e400/0x678ae32e95329477 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efb7b expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695684.422317:0:100468:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695684.423509:0:100468:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787695684.423532:0:100468:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787695684.423544:0:100468:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd7480 x1874531134107520/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787695684.423552:0:100468:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695684.426241:0:100468:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107520:192.168.201.137@tcp:101:touch.0 00000100:00100000:0.0:1787695684.428145:0:100468:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8bc00 going to sleep for 12 seconds 00000100:00100000:0.0:1787695684.434172:0:100468:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107520:192.168.201.137@tcp:101:touch.0 00010000:00010000:0.0:1787695684.434201:0:100468:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434235:0:100468:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 4/0,1 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434243:0:100468:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 4/0,1 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434256:0:100468:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434266:0:100468:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434278:0:100468:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434285:0:100468:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434291:0:100468:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695684.434302:0:100468:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9d85babd7480 x1874531134107520/t4294997969(4294997969) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695701 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787695684.434313:0:100468:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85babd7480 x1874531134107520/t4294997969(4294997969) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695701 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695684.434325:0:100468:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787695684.434342:0:100468:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434352:0:100468:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589935617/1393 (0 0) 00000080:00200000:0.0:1787695684.434358:0:100468:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695684.434372:0:100468:0:(namei.c:1628:ll_create_it()) VFS Op:name=f43c.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000080:00200000:0.0:1787695684.434430:0:100468:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x571:0x0](ffff9d85b3d17448) 00000080:00200000:0.0:1787695684.434505:0:100468:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x571:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787695684.434510:0:100468:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) 00000080:00010000:0.0:1787695684.434513:0:100468:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) for lock 0x678ae32e9532947e 00000080:00010000:0.0:1787695684.434523:0:100468:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) 00000080:00200008:0.0:1787695684.434537:0:100468:0:(file.c:1090:ll_file_open()) START file f43c.sanityn-2:[0x200000401:0x571:0x0](ffff9d85b3d17448), flags 104501 00000002:00100000:0.0:1787695684.434553:0:100468:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85babd7480 x1874531134107520/t4294997969(4294997969) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695701 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787695684.434584:0:100468:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532947e from it ffff9d8587b876c0 00010000:00010000:0.0:1787695684.434589:0:100468:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.434596:0:100468:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695684.434616:0:100468:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000401:0x571:0x0](ffff9d85b3d17448), inode mode 81a4 mask 40 00000080:00200008:0.0:1787695684.434651:0:100468:0:(llite_lib.c:2427:ll_setattr_raw()) START file f43c.sanityn-2:[0x200000401:0x571:0x0](ffff9d85b3d17448) current size 0, valid attrs 20070, mode 9d85, uid 29982288, gid -18988, new size 18446744072543992343, atime 1787695684.000000000, mtime 1787695684.000000000, ctime 1787695684.000000000 00010000:00010000:0.0:1787695684.434670:0:100468:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695684.434685:0:100468:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x571:0x0]: losing layout lock 00010000:00010000:0.0:1787695684.434719:0:100468:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 1/0,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695684.434726:0:100468:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695684.434728:0:100468:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86fa00/0x678ae32e9532947e lrc: 0/0,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efb97 expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787695684.434746:0:100468:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695684.434753:0:100468:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d0b800 x1874531134107776/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695684.434773:0:100468:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107776:192.168.201.137@tcp:36:touch.0 00000100:00100000:0.0:1787695684.434831:0:100468:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9eccf00 going to sleep for 12 seconds 00000100:00100000:0.0:1787695684.462976:0:100468:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107776:192.168.201.137@tcp:36:touch.0 00000080:00200000:0.0:1787695684.463031:0:100468:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x571:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787695684.463052:0:100468:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000401:0x571:0x0](ffff9d85b3d17448) 00000002:00010000:0.0:1787695684.463062:0:100468:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x571:0x0]) in obj [0x200000401:0x571:0x0], intent: layout flags 0 00010000:00010000:0.0:1787695684.463113:0:100468:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x571:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.463123:0:100468:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.463201:0:100468:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 100468 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695684.463223:0:100468:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107904:192.168.201.137@tcp:101:touch.0 00000100:00100000:0.0:1787695684.463291:0:100468:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9eccf00 going to sleep for 12 seconds 00000100:00100000:0.0:1787695684.469837:0:100468:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:100468:1874531134107904:192.168.201.137@tcp:101:touch.0 00010000:00010000:0.0:1787695684.470286:0:100468:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.470303:0:100468:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695684.470320:0:100468:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9d85b9d09500 x1874531134107904/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 376/400 e 0 to 0 dl 1787695701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695684.470343:0:100468:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787695684.470357:0:100468:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.470370:0:100468:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 16 res 8589935617/1393 (0 0) 00000080:00200000:0.0:1787695684.470717:0:100468:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000401:0x571:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787695684.470737:0:100468:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) for lock 0x678ae32e95329485 00000080:00010000:0.0:1787695684.470746:0:100468:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329485 from it ffffb5d401c97a78 00010000:00010000:0.0:1787695684.470751:0:100468:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695684.470762:0:100468:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695684.470788:0:100468:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695684.470799:0:100468:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787695684.470808:0:100468:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000401:0x571:0x0](00000000937280fd) being reconfigured ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695684.471263:0:100468:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x571:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1787695684.471271:0:100468:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695684.471280:0:100468:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898da00/0x678ae32e95329485 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x571:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efbac expref: -99 pid: 100468 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787695684.471303:0:100468:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1787695684.477034:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:68e474e8-42cb-4207-86a3-041065654492:2361:1874531134108032:192.168.201.137@tcp:2:touch.0 00010000:00010000:2.0:1787695685.843097:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1787695686.177399:0:100530:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695686.177417:0:100530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532948c 00000080:00010000:2.0:1787695686.177422:0:100530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695686.177425:0:100530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532948c from it ffffb5d401f2fb58 00010000:00010000:2.0:1787695686.177429:0:100530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.177437:0:100530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695686.177464:0:100530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695686.177485:0:100530:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f43c.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=8 00000080:00200000:2.0:1787695686.177489:0:100530:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f43c.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=getattr 00000002:00010000:2.0:1787695686.177509:0:100530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f43c.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695686.177551:0:100530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 100530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.177558:0:100530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 100530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.177566:0:100530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 100530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695686.177587:0:100530:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100530:1874531134108544:192.168.201.137@tcp:101:rm.0 00000100:00100000:2.0:1787695686.177683:0:100530:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8b0c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695686.198717:0:100530:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100530:1874531134108544:192.168.201.137@tcp:101:rm.0 00010000:00010000:2.0:1787695686.198759:0:100530:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198775:0:100530:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198783:0:100530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198795:0:100530:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198806:0:100530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198818:0:100530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198831:0:100530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198838:0:100530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695686.198902:0:100530:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd6680 x1874531134108544/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/640 e 0 to 0 dl 1787695703 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695686.198915:0:100530:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695686.198928:0:100530:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.198939:0:100530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589935617/1392 (0 0) 00000080:00200000:2.0:1787695686.198946:0:100530:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695686.198993:0:100530:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x570:0x0](ffff9d85b3d0ec08): rc = 0 00000080:00200000:2.0:1787695686.199007:0:100530:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x570:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787695686.199012:0:100530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) for lock 0x678ae32e95329493 00000080:00010000:2.0:1787695686.199025:0:100530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) 00000080:00010000:2.0:1787695686.199028:0:100530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329493 from it ffffb5d401f2fb20 00010000:00010000:2.0:1787695686.199032:0:100530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.199040:0:100530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787695686.199084:0:100530:0:(file.c:6296:ll_getattr_dentry()) START file f43c.sanityn:[0x200000401:0x570:0x0](ffff9d85b3d0ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787695686.199091:0:100530:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x570:0x0](ffff9d85b3d0ec08),name=f43c.sanityn 00000002:00010000:2.0:1787695686.199110:0:100530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x570:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695686.199119:0:100530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695686.199132:0:100530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695686.199139:0:100530:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695686.199144:0:100530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) 00000080:00010000:2.0:1787695686.199148:0:100530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x570:0x0](ffff9d85b3d0ec08) for lock 0x678ae32e95329493 00000080:00010000:2.0:1787695686.199153:0:100530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329493 from it ffffb5d401f2fcb8 00010000:00010000:2.0:1787695686.199156:0:100530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695686.199163:0:100530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695686.199186:0:100530:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x570:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787695686.199198:0:100530:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x570:0x0] 00000008:00010000:2.0:1787695686.199228:0:100530:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787695686.199232:0:100530:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9d858345c0b0, osc lock ffff9d858350ce40, flags 0x400001002 00010000:00010000:2.0:1787695686.199291:0:100530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 000000001d9f97b7/0x678ae32e9532949a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x821:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 100530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695686.199318:0:100530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 000000001d9f97b7/0x678ae32e9532949a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x821:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 100530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787695686.199364:0:100530:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1787695686.211112:0:100530:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x570:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787695686.211120:0:100530:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8588a57c20 (ffff9d85a93eba28/0) at cl_lock_release():245 00000020:00010000:2.0:1787695686.211123:0:100530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8588a57c20 (ffff9d85a93eba28/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787695686.211129:0:100530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d858345c0b0 (ffff9d85a91d9ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787695686.211133:0:100530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8588a57c20 (ffff9d85a93eba28/0) at cl_lock_fini():72 00000020:00010000:2.0:1787695686.211136:0:100530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d858345c0b0 (ffff9d85a93eba28/1) at cl_lock_fini():72 00000080:00200000:2.0:1787695686.211150:0:100530:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x570:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787695686.211210:0:100530:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695686.211346:0:100530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695686.211364:0:100530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695686.211377:0:100530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695686.211384:0:100530:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695686.211393:0:100530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695686.211396:0:100530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532948c 00000080:00010000:2.0:1787695686.211464:0:100530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532948c from it ffffb5d401f2fc10 00010000:00010000:2.0:1787695686.211469:0:100530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695686.211483:0:100530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695686.211494:0:100530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695686.211504:0:100530:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f43c.sanityn, flags=0 00000080:00200000:2.0:1787695686.211509:0:100530:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695686.211516:0:100530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695686.211522:0:100530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695686.211531:0:100530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695686.211536:0:100530:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695686.211541:0:100530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695686.211543:0:100530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532948c 00000080:00010000:2.0:1787695686.211547:0:100530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532948c from it ffffb5d401f2fd68 00010000:00010000:2.0:1787695686.211550:0:100530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695686.211557:0:100530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695686.211563:0:100530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695686.211570:0:100530:0:(namei.c:2418:ll_unlink()) VFS Op:name=f43c.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695686.211653:0:100530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695686.211682:0:100530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695686.211693:0:100530:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x570:0x0]: losing layout lock 00010000:00010000:2.0:1787695686.211704:0:100530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695686.211713:0:100530:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x570:0x0]: losing layout lock 00010000:00010000:2.0:1787695686.212075:0:100530:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695686.212084:0:100530:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 1/0,0 mode: --/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695686.212090:0:100530:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695686.212096:0:100530:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1787695686.212099:0:100530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f800/0x678ae32e95329493 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x570:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efbd6 expref: -99 pid: 100530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695686.212113:0:100530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86f200/0x678ae32e95329470 lrc: 0/0,0 mode: --/CR res: [0x200000401:0x570:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efb66 expref: -99 pid: 100467 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695686.212125:0:100530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031400/0x678ae32e9532948c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efbba expref: -99 pid: 100530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695686.212146:0:100530:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695686.212157:0:100530:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd5c00 x1874531134108928/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695686.212179:0:100530:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100530:1874531134108928:192.168.201.137@tcp:36:rm.0 00000100:00100000:2.0:1787695686.212674:0:100530:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8b0c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695688.322562:0:100530:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d8590d43000: committing for last_committed 4294997971 gen 1 00000100:00100000:2.0:1787695688.322630:0:100530:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9d85babd5c00 x1874531134108928/t4294997972(4294997972) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 504/456 e 0 to 0 dl 1787695703 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695688.322647:0:100530:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100530:1874531134108928:192.168.201.137@tcp:36:rm.0 00000080:00200000:2.0:1787695688.322707:0:100530:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x570:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787695688.322721:0:100530:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787695688.322751:0:100530:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x570:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787695688.322759:0:100530:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x570:0x0](ffff9d85b3d0ec08) 00000008:00010000:2.0:1787695688.322795:0:100530:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1787695684 1787695684 1787695684 by oinfo size 0 blocks 1 [cma]time 1787695684 1787695684 1787695684 ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 000000001d9f97b7/0x678ae32e9532949a lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x821:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x297ceef0832efbdd expref: -99 pid: 100530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1787695688.336068:0:100532:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695688.336085:0:100532:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e953294a1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbf9 expref: -99 pid: 100532 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1787695688.336097:0:100532:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e953294a1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbf9 expref: -99 pid: 100532 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1787695688.336103:0:100532:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695688.336110:0:100532:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) 00000080:00010000:2.0:1787695688.336113:0:100532:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad8db88) for lock 0x678ae32e953294a1 00000080:00010000:2.0:1787695688.336119:0:100532:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294a1 from it ffffb5d401f47d90 00010000:00010000:2.0:1787695688.336123:0:100532:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e953294a1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efbf9 expref: -99 pid: 100532 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1787695688.336132:0:100532:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e953294a1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efbf9 expref: -99 pid: 100532 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1787695688.336145:0:100532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88), inode mode 41ff mask 3 00000080:00200000:2.0:1787695688.336153:0:100532:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x571:0x0](ffff9d85b3cef448), dir=[0x200000007:0x1:0x0](ffff9d858ad8db88), target=f43c.sanityn 00010000:00010000:2.0:1787695688.336168:0:100532:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e953294a1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efbf9 expref: -99 pid: 100532 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1787695688.336191:0:100532:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e953294a8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efc23 expref: -99 pid: 100532 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695688.336206:0:100532:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x571:0x0]: losing layout lock 00010000:00010000:2.0:1787695688.336238:0:100532:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e953294a8 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efc23 expref: -99 pid: 100532 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695688.336245:0:100532:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e953294a1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efbf9 expref: -99 pid: 100532 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1787695688.336252:0:100532:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695688.336255:0:100532:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936f000/0x678ae32e953294a8 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efc23 expref: -99 pid: 100532 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695688.336441:0:100532:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d859936fc00/0x678ae32e953294a1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efbf9 expref: -99 pid: 100532 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00100000:2.0:1787695688.336468:0:100532:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d858c4eb000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695688.336481:0:100532:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd7480 x1874531134109440/t0(0) o36->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695688.336506:0:100532:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job link:b3d8e19c-573a-445a-9e98-5fe67f1b3715:100532:1874531134109440:192.168.201.137@tcp:36:link.0 00000100:00100000:2.0:1787695688.336773:0:100532:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8b900 going to sleep for 12 seconds 00000100:00100000:2.0:1787695688.356649:0:100532:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job link:b3d8e19c-573a-445a-9e98-5fe67f1b3715:100532:1874531134109440:192.168.201.137@tcp:36:link.0 00000080:00200000:2.0:1787695688.971759:0:100573:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695688.971792:0:100573:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695688.971823:0:100573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031e00/0x678ae32e953294af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc46 expref: -99 pid: 100221 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695688.971838:0:100573:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031e00/0x678ae32e953294af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc46 expref: -99 pid: 100221 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695688.971846:0:100573:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695688.971880:0:100573:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695688.971884:0:100573:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953294af 00000080:00010000:2.0:1787695688.971892:0:100573:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294af from it ffffb5d401f57b58 00010000:00010000:2.0:1787695688.971897:0:100573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031e00/0x678ae32e953294af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efc46 expref: -99 pid: 100221 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695688.971905:0:100573:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031e00/0x678ae32e953294af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc46 expref: -99 pid: 100221 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787695688.971922:0:100573:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695688.971935:0:100573:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f43c.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags=8 00000080:00200000:2.0:1787695688.971939:0:100573:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f43c.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=getattr 00000002:00010000:2.0:1787695688.971951:0:100573:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f43c.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787695688.971994:0:100573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294b6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 100573 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.972002:0:100573:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294b6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 100573 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.972011:0:100573:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294b6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 100573 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695688.972037:0:100573:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100573:1874531134109824:192.168.201.137@tcp:101:rm.0 00000100:00100000:2.0:1787695688.973491:0:100573:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8b900 going to sleep for 12 seconds 00000100:00100000:2.0:1787695688.982520:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134109952:192.168.201.137@tcp:101:rm.0 00000100:00100000:2.0:1787695688.988466:0:2362:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d8590d43000: committing for last_committed 4294969647 gen 16 00010000:00010000:2.0:1787695688.988500:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.988521:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.988536:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.988547:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787695688.990530:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787695688.990548:0:2362:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1787695688.990639:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.990649:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.990656:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695688.990665:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1787695688.990688:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134109952:192.168.201.137@tcp:101:rm.0 00000100:00100000:2.0:1787695689.016902:0:100573:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100573:1874531134110464:192.168.201.137@tcp:101:rm.0 00000100:00100000:2.0:1787695689.017023:0:100573:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8b0c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695689.030223:0:100573:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100573:1874531134110464:192.168.201.137@tcp:101:rm.0 00010000:00010000:2.0:1787695689.030252:0:100573:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030291:0:100573:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030299:0:100573:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030310:0:100573:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030320:0:100573:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030340:0:100573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030348:0:100573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030355:0:100573:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695689.030366:0:100573:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9ce0380 x1874531134110464/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/640 e 0 to 0 dl 1787695706 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695689.030376:0:100573:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695689.030393:0:100573:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030402:0:100573:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589935617/1393 (0 0) 00000080:00200000:2.0:1787695689.030408:0:100573:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695689.030452:0:100573:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x571:0x0](ffff9d85b3d17448): rc = 0 00000080:00200000:2.0:1787695689.030466:0:100573:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x571:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1787695689.030472:0:100573:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) for lock 0x678ae32e953294cb 00000080:00010000:2.0:1787695689.030481:0:100573:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) 00000080:00010000:2.0:1787695689.030484:0:100573:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294cb from it ffffb5d401f57b20 00010000:00010000:2.0:1787695689.030488:0:100573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695689.030496:0:100573:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787695689.030533:0:100573:0:(file.c:6296:ll_getattr_dentry()) START file f43c.sanityn-2:[0x200000401:0x571:0x0](ffff9d85b3d17448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787695689.030539:0:100573:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x571:0x0](ffff9d85b3d17448),name=f43c.sanityn-2 00000002:00010000:2.0:1787695689.030748:0:100573:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x571:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695689.030761:0:100573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695689.030773:0:100573:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695689.030780:0:100573:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695689.030786:0:100573:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) 00000080:00010000:2.0:1787695689.030790:0:100573:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x571:0x0](ffff9d85b3d17448) for lock 0x678ae32e953294cb 00000080:00010000:2.0:1787695689.030794:0:100573:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294cb from it ffffb5d401f57cb8 00010000:00010000:2.0:1787695689.030807:0:100573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695689.030815:0:100573:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695689.030836:0:100573:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x571:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787695689.030845:0:100573:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x571:0x0] 00000008:00010000:2.0:1787695689.030916:0:100573:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787695689.030921:0:100573:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9d858345c0b0, osc lock ffff9d858350ce40, flags 0x400001002 00010000:00010000:2.0:1787695689.030941:0:100573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695689.030964:0:100573:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1787695689.030977:0:100573:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787695689.030980:0:100573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695689.030988:0:100573:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1787695689.030998:0:100573:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787695689.031010:0:100573:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x571:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787695689.031014:0:100573:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8588a57c20 (ffff9d85a93eba28/0) at cl_lock_release():245 00000020:00010000:2.0:1787695689.031017:0:100573:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8588a57c20 (ffff9d85a93eba28/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787695689.031020:0:100573:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d858345c0b0 (ffff9d85a91d9ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787695689.031024:0:100573:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8588a57c20 (ffff9d85a93eba28/0) at cl_lock_fini():72 00000020:00010000:2.0:1787695689.031027:0:100573:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d858345c0b0 (ffff9d85a93eba28/1) at cl_lock_fini():72 00000080:00200000:2.0:1787695689.031040:0:100573:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x571:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787695689.031084:0:100573:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695689.031093:0:100573:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695689.031100:0:100573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695689.031108:0:100573:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695689.031114:0:100573:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695689.031118:0:100573:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695689.031121:0:100573:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953294c4 00000080:00010000:2.0:1787695689.031124:0:100573:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294c4 from it ffffb5d401f57c10 00010000:00010000:2.0:1787695689.031128:0:100573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695689.031135:0:100573:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695689.031143:0:100573:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695689.031152:0:100573:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f43c.sanityn-2, flags=0 00000080:00200000:2.0:1787695689.031157:0:100573:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695689.031164:0:100573:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695689.031169:0:100573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695689.031177:0:100573:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695689.031183:0:100573:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695689.031187:0:100573:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695689.031190:0:100573:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953294c4 00000080:00010000:2.0:1787695689.031193:0:100573:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294c4 from it ffffb5d401f57d68 00010000:00010000:2.0:1787695689.031196:0:100573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695689.031203:0:100573:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695689.031211:0:100573:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695689.031216:0:100573:0:(namei.c:2418:ll_unlink()) VFS Op:name=f43c.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:2.0:1787695689.031237:0:100573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695689.031499:0:100573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695689.031511:0:100573:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x571:0x0]: losing layout lock 00010000:00010000:2.0:1787695689.031545:0:100573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 1/0,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695689.031557:0:100573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695689.031639:0:100573:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787695689.031642:0:100573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936f600/0x678ae32e953294cb lrc: 0/0,0 mode: --/PR res: [0x200000401:0x571:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efca8 expref: -99 pid: 100573 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695689.031664:0:100573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936ea00/0x678ae32e953294c4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efc8c expref: -99 pid: 100573 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1787695689.031686:0:100573:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695689.031696:0:100573:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd4000 x1874531134110720/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695689.031722:0:100573:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100573:1874531134110720:192.168.201.137@tcp:36:rm.0 00000100:00100000:2.0:1787695689.031834:0:100573:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d8589f8b0c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695689.044949:0:100573:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:100573:1874531134110720:192.168.201.137@tcp:36:rm.0 00000080:00200000:2.0:1787695689.045021:0:100573:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x571:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787695689.045037:0:100573:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787695689.045069:0:100573:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x571:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787695689.045079:0:100573:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x571:0x0](ffff9d85b3d17448) 00000008:00010000:2.0:1787695689.045116:0:100573:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1787695684 1787695684 1787695684 by oinfo size 0 blocks 1 [cma]time 1787695684 1787695684 1787695684 ns: lustre-OST0001-osc-ffff9d8590d43000 lock: 00000000337497a2/0x678ae32e953294bd lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x820:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x297ceef0832efc69 expref: -99 pid: 100573 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787695689.363765:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2363:1874531134110848:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695689.363926:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134111104:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695689.363970:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134111360:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695689.363995:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134111616:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695689.364028:0:2363:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1787695689.364034:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134110976:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695689.364060:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134111232:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695689.364083:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134111488:192.168.201.137@tcp:400:kworker.0 00010000:00010000:0.0:1787695689.952734:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695691.992876:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695692.970889:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695694.046190:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787695694.485096:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2360:1874531134112000:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.485211:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134112256:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.485254:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134112512:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.485290:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134112768:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.485322:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:0.0:1787695694.485328:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134112128:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.485349:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134112384:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.485371:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134112640:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.492343:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d8590d43000: committing for last_committed 4294997975 gen 1 00000100:00100000:0.0:1787695694.492388:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134112128:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.492404:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134112384:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.492416:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134112640:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.492427:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2360:1874531134112000:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.492438:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:68e474e8-42cb-4207-86a3-041065654492:2360:1874531134112256:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.492448:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d858c4eb000: committing for last_committed 4294997973 gen 1 00000100:00100000:0.0:1787695694.492457:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134112512:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695694.492468:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134112768:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695695.063872:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695696.083607:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695697.107836:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695697.371280:0:101140:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9d858c4eb000 00010000:00010000:0.0:1787695697.371341:0:101140:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9d8590d43000 00000100:00100000:0.0:1787695699.604702:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2361:1874531134112896:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695699.605047:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787695699.605056:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:68e474e8-42cb-4207-86a3-041065654492:2361:1874531134113024:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695699.649226:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2360:1874531134113536:192.168.201.137@tcp:400:kworker.0 00000080:00200000:2.0:1787695700.040266:0:101245:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695700.040298:0:101245:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695700.040362:0:101245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.040374:0:101245:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.040385:0:101245:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695700.040412:0:101245:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134113792:192.168.201.137@tcp:101:touch.0 00000100:00100000:2.0:1787695700.040499:0:101245:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b72d8d80 going to sleep for 12 seconds 00000100:00100000:2.0:1787695700.051963:0:101245:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134113792:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695700.051992:0:101245:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052006:0:101245:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052018:0:101245:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052029:0:101245:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052041:0:101245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052050:0:101245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052067:0:101245:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695700.052080:0:101245:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85babd7800 x1874531134113792/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695717 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695700.052094:0:101245:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052114:0:101245:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1787695700.052121:0:101245:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787695700.052148:0:101245:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1787695700.052153:0:101245:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953294d2 00000080:00010000:2.0:1787695700.052159:0:101245:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695700.052162:0:101245:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294d2 from it ffffb5d401d4fbf8 00010000:00010000:2.0:1787695700.052174:0:101245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052184:0:101245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695700.052208:0:101245:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1787695700.052229:0:101245:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695700.052243:0:101245:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695700.052251:0:101245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695700.052261:0:101245:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695700.052267:0:101245:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695700.052271:0:101245:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695700.052274:0:101245:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953294d2 00000080:00010000:2.0:1787695700.052278:0:101245:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294d2 from it ffffb5d401d4fb38 00010000:00010000:2.0:1787695700.052281:0:101245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695700.052288:0:101245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695700.052296:0:101245:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1787695700.052301:0:101245:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f43d.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), file ffff9d8584666900, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1787695700.052309:0:101245:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f43d.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000002:00010000:2.0:1787695700.052328:0:101245:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f43d.sanityn,[0x200000401:0x572:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1787695700.052339:0:101245:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695700.052397:0:101245:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695700.052404:0:101245:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695700.052407:0:101245:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294d2 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efce0 expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695700.052441:0:101245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.052449:0:101245:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787695700.052460:0:101245:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787695700.052467:0:101245:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd7800 x1874531134114048/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787695700.052476:0:101245:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695700.052494:0:101245:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114048:192.168.201.137@tcp:101:touch.0 00000100:00100000:2.0:1787695700.052827:0:101245:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b72d8d80 going to sleep for 12 seconds 00000100:00100000:2.0:1787695700.061633:0:101245:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114048:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695700.061665:0:101245:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061696:0:101245:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 4/0,1 mode: --/PR res: [0x200000401:0x572:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061706:0:101245:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 4/0,1 mode: --/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061718:0:101245:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061728:0:101245:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061741:0:101245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061750:0:101245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061758:0:101245:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695700.061771:0:101245:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9d85babd7800 x1874531134114048/t4294997976(4294997976) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695717 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787695700.061785:0:101245:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85babd7800 x1874531134114048/t4294997976(4294997976) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695717 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695700.061799:0:101245:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695700.061811:0:101245:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.061822:0:101245:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589935617/1394 (0 0) 00000080:00200000:2.0:1787695700.061828:0:101245:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787695700.061840:0:101245:0:(namei.c:1628:ll_create_it()) VFS Op:name=f43d.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000080:00200000:2.0:1787695700.061896:0:101245:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x572:0x0](ffff9d85b3d0ec08) 00000080:00200000:2.0:1787695700.061982:0:101245:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x572:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1787695700.061988:0:101245:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000401:0x572:0x0](ffff9d85b3d0ec08) 00000080:00010000:2.0:1787695700.061991:0:101245:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x572:0x0](ffff9d85b3d0ec08) for lock 0x678ae32e953294d9 00000080:00010000:2.0:1787695700.062001:0:101245:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x572:0x0](ffff9d85b3d0ec08) 00000080:00200008:2.0:1787695700.062015:0:101245:0:(file.c:1090:ll_file_open()) START file f43d.sanityn:[0x200000401:0x572:0x0](ffff9d85b3d0ec08), flags 104501 00000002:00100000:2.0:1787695700.062032:0:101245:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85babd7800 x1874531134114048/t4294997976(4294997976) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695717 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787695700.062069:0:101245:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294d9 from it ffff9d8587b879c0 00010000:00010000:2.0:1787695700.062074:0:101245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.062082:0:101245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695700.062111:0:101245:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000401:0x572:0x0](ffff9d85b3d0ec08), inode mode 81a4 mask 40 00000080:00200008:2.0:1787695700.062152:0:101245:0:(llite_lib.c:2427:ll_setattr_raw()) START file f43d.sanityn:[0x200000401:0x572:0x0](ffff9d85b3d0ec08) current size 0, valid attrs 20070, mode 9d85, uid 30735952, gid -18988, new size 18446744072543992343, atime 1787695700.000000000, mtime 1787695700.000000000, ctime 1787695700.000000000 00010000:00010000:2.0:1787695700.062524:0:101245:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695700.062545:0:101245:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x572:0x0]: losing layout lock 00010000:00010000:2.0:1787695700.062648:0:101245:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695700.062656:0:101245:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695700.062658:0:101245:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ee00/0x678ae32e953294d9 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efcfc expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787695700.062680:0:101245:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787695700.062687:0:101245:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd4380 x1874531134114304/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695700.062713:0:101245:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114304:192.168.201.137@tcp:36:touch.0 00000100:00100000:2.0:1787695700.062796:0:101245:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b72d83c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695700.067447:0:101245:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114304:192.168.201.137@tcp:36:touch.0 00000080:00200000:2.0:1787695700.067502:0:101245:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x572:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1787695700.067523:0:101245:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000401:0x572:0x0](ffff9d85b3d0ec08) 00000002:00010000:2.0:1787695700.067535:0:101245:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x572:0x0]) in obj [0x200000401:0x572:0x0], intent: layout flags 0 00010000:00010000:2.0:1787695700.067584:0:101245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x572:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.067595:0:101245:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.067621:0:101245:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101245 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787695700.067641:0:101245:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114432:192.168.201.137@tcp:101:touch.0 00000100:00100000:2.0:1787695700.067728:0:101245:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b72d83c0 going to sleep for 12 seconds 00000100:00100000:2.0:1787695700.071088:0:101245:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114432:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695700.071131:0:101245:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.071149:0:101245:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695700.071174:0:101245:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9d85babd6d80 x1874531134114432/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 376/400 e 0 to 0 dl 1787695717 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695700.071187:0:101245:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695700.071201:0:101245:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.071216:0:101245:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 16 res 8589935617/1394 (0 0) 00000080:00200000:2.0:1787695700.071224:0:101245:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000401:0x572:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787695700.071245:0:101245:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x572:0x0](ffff9d85b3d0ec08) for lock 0x678ae32e953294e0 00000080:00010000:2.0:1787695700.071256:0:101245:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294e0 from it ffffb5d401d4fa78 00010000:00010000:2.0:1787695700.071260:0:101245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695700.071271:0:101245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695700.071295:0:101245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695700.071307:0:101245:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787695700.071315:0:101245:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000401:0x572:0x0](00000000e7ece8ff) being reconfigured ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695700.071495:0:101245:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x572:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1787695700.071505:0:101245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695700.071513:0:101245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787695700.071536:0:101245:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787695700.075027:0:101245:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x572:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787695700.075043:0:101245:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f43d.sanityn:[0x200000401:0x572:0x0](ffff9d85b3d0ec08) current size 0, valid attrs 100201f0, mode 9d85, uid 30735952, gid -18988, new size 18446744072543992343, atime 1787695700.000000000, mtime 1787695700.000000000, ctime 1787695700.000000000, rc 0 00000080:00200008:2.0:1787695700.075072:0:101245:0:(file.c:479:ll_file_release()) START file f43d.sanityn:[0x200000401:0x572:0x0](ffff9d85b3d0ec08), flags 104001 00000002:00100000:2.0:1787695700.075146:0:101245:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d85babd7800 x1874531134114048/t4294997976(4294997976) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695717 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787695700.075159:0:101245:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787695700.075273:0:101245:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85babd5880 x1874531134114816/t0(0) o35->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787695700.075298:0:101245:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5880 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114816:192.168.201.137@tcp:35:touch.0 00000100:00100000:2.0:1787695700.075442:0:101245:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b72d83c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787695700.079102:0:101245:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5880 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:101245:1874531134114816:192.168.201.137@tcp:35:touch.0 00000002:00100000:2.0:1787695700.079150:0:101245:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d85babd7800 x1874531134114048/t4294997976(4294997976) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695717 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787695700.179164:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695701.203295:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787695701.526325:0:101306:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695701.526354:0:101306:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695701.526417:0:101306:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 101306 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695701.526428:0:101306:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101306 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695701.526438:0:101306:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101306 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695701.526462:0:101306:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:101306:1874531134114944:192.168.201.137@tcp:101:rm.0 00000100:00100000:0.0:1787695701.526713:0:101306:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85879ab300 going to sleep for 12 seconds 00000080:00200000:0.0:1787695701.554412:0:101306:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x572:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1787695701.554419:0:101306:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d85b7257c20 (ffff9d85baa29758/0) at cl_lock_release():245 00000020:00010000:0.0:1787695701.554423:0:101306:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d85b7257c20 (ffff9d85baa29758/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787695701.554428:0:101306:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d85b727ea30 (ffff9d85baa293f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787695701.554432:0:101306:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d85b7257c20 (ffff9d85baa29758/0) at cl_lock_fini():72 00000020:00010000:0.0:1787695701.554435:0:101306:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d85b727ea30 (ffff9d85baa29758/1) at cl_lock_fini():72 00000080:00200000:0.0:1787695701.554453:0:101306:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x572:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787695701.554518:0:101306:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695701.554538:0:101306:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695701.554555:0:101306:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695701.554574:0:101306:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695701.554592:0:101306:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695701.554600:0:101306:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695701.554604:0:101306:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953294e7 00000080:00010000:0.0:1787695701.554611:0:101306:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294e7 from it ffffb5d401f17c10 00010000:00010000:0.0:1787695701.554615:0:101306:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695701.554622:0:101306:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695701.554632:0:101306:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695701.554642:0:101306:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f43d.sanityn, flags=0 00000080:00200000:0.0:1787695701.554649:0:101306:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695701.554655:0:101306:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695701.554662:0:101306:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695701.554671:0:101306:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695701.554677:0:101306:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695701.554680:0:101306:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695701.554682:0:101306:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e953294e7 00000080:00010000:0.0:1787695701.554686:0:101306:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e953294e7 from it ffffb5d401f17d68 00010000:00010000:0.0:1787695701.554690:0:101306:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695701.554696:0:101306:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787695701.554705:0:101306:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695701.554710:0:101306:0:(namei.c:2418:ll_unlink()) VFS Op:name=f43d.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08) 00010000:00010000:0.0:1787695701.554739:0:101306:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695701.554780:0:101306:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e400/0x678ae32e953294ee lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efd3b expref: -99 pid: 101306 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695701.554792:0:101306:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x572:0x0]: losing layout lock 00010000:00010000:0.0:1787695701.554805:0:101306:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695701.554813:0:101306:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x572:0x0]: losing layout lock 00010000:00010000:0.0:1787695701.554838:0:101306:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e400/0x678ae32e953294ee lrc: 1/0,0 mode: --/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efd3b expref: -99 pid: 101306 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695701.554846:0:101306:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 1/0,0 mode: --/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695701.554897:0:101306:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787695701.554903:0:101306:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1787695701.554907:0:101306:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d859936e400/0x678ae32e953294ee lrc: 0/0,0 mode: --/PR res: [0x200000401:0x572:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efd3b expref: -99 pid: 101306 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695701.554920:0:101306:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d85ae86ec00/0x678ae32e953294e0 lrc: 0/0,0 mode: --/CR res: [0x200000401:0x572:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efd11 expref: -99 pid: 101245 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695701.554935:0:101306:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e953294e7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efd1f expref: -99 pid: 101306 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787695701.554953:0:101306:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695701.554961:0:101306:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d0a680 x1874531134115456/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695701.554985:0:101306:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job rm:68e474e8-42cb-4207-86a3-041065654492:101306:1874531134115456:192.168.201.137@tcp:36:rm.0 00000100:00100000:0.0:1787695701.555081:0:101306:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85879abcc0 going to sleep for 12 seconds 00000080:00200000:0.0:1787695702.075162:0:101308:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad8db88),name=/ 00000002:00010000:0.0:1787695702.075199:0:101308:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695702.075268:0:101308:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e953294fc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 101308 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695702.075279:0:101308:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e953294fc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101308 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695702.075290:0:101308:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e953294fc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 101308 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695702.075317:0:101308:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job rm:b3d8e19c-573a-445a-9e98-5fe67f1b3715:101308:1874531134115584:192.168.201.137@tcp:101:rm.0 00000100:00100000:0.0:1787695702.075425:0:101308:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85879ab840 going to sleep for 12 seconds 00010000:00010000:2.0:1787695702.228207:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787695704.647766:0:101000:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job bash:68e474e8-42cb-4207-86a3-041065654492:101000:1874531134115968:192.168.201.137@tcp:37:bash.0 00000100:00100000:0.0:1787695704.648006:0:101000:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85879abc00 going to sleep for 11 seconds 00000100:00100000:2.0:1787695704.664010:0:101000:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job bash:68e474e8-42cb-4207-86a3-041065654492:101000:1874531134115968:192.168.201.137@tcp:37:bash.0 00010000:00010000:2.0:1787695704.666487:0:101000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695704.666512:0:101000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695704.666558:0:101000:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:2.0:1787695704.666582:0:101000:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787695704.666596:0:101000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695704.666609:0:101000:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787695704.666617:0:101000:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787695704.666623:0:101000:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:2.0:1787695704.666626:0:101000:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e9532950a 00000080:00010000:2.0:1787695704.666632:0:101000:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532950a from it ffffb5d400d87cd8 00010000:00010000:2.0:1787695704.666635:0:101000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787695704.666642:0:101000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787695704.666873:0:101000:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08) pos/size0/11776 32bit_api 0 00000080:00200000:2.0:1787695704.666957:0:101000:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200008:2.0:1787695704.666978:0:101000:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9d858ad3ec08), flags 304000 00000100:00100000:2.0:1787695704.723681:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134116352:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695704.723796:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134116608:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695704.723822:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134116864:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695704.723874:0:2363:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1787695704.723880:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2363:1874531134116224:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695704.723906:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134116480:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695704.723924:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134116736:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695704.739791:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2363:1874531134116224:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695704.739827:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134116608:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695705.299254:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695706.324143:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787695708.316053:0:2352:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.137@tcp (source: 12345-192.168.201.137@tcp) 00000100:00100000:0.0:1787695708.316101:0:5589:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874531140570624 00000100:00100000:0.0:1787695708.316111:0:5589:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874531140570624 00000100:00100000:0.0:1787695708.316161:0:5589:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.137@tcp, seq: 10506 00000100:00100000:0.0:1787695708.316172:0:5589:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9d85b9d08380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6687:x1874531140570624:12345-192.168.201.137@tcp:104: 00010000:00010000:0.0:1787695708.316191:0:5589:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 00000000ebd520a6/0x678ae32e953294f5 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x822:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x297ceef0832efd42 expref: -99 pid: 101306 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695708.316262:0:5589:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787695708.316283:0:5589:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9d85b9d08380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6687:x1874531140570624:12345-192.168.201.137@tcp:104: Request processed in 100us (226us total) trans 0 rc 0/0 00000100:00100000:0.0:1787695708.316292:0:5589:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.137@tcp, seq: 10506 00010000:00010000:2.0:1787695708.322078:0:14499:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787695708.322095:0:14499:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 00000000ebd520a6/0x678ae32e953294f5 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x822:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x297ceef0832efd42 expref: -99 pid: 101306 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695708.322110:0:14499:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d859936fe00 already unused, calling callback (ffffffffc12f4300) 00010000:00010000:2.0:1787695708.322118:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 00000000ebd520a6/0x678ae32e953294f5 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x822:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x297ceef0832efd42 expref: -99 pid: 101306 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695708.322179:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 00000000ebd520a6/0x678ae32e953294f5 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x822:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x297ceef0832efd42 expref: -99 pid: 101306 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695708.322187:0:14499:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787695708.322215:0:14499:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 00000000ebd520a6/0x678ae32e953294f5 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x822:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x297ceef0832efd42 expref: -99 pid: 101306 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695708.322223:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d8590d43000 lock: 00000000ebd520a6/0x678ae32e953294f5 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x822:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x297ceef0832efd42 expref: -99 pid: 101306 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695708.322249:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695708.322251:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695708.322253:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695708.374355:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695709.396370:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695711.446148:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695712.765570:0:14499:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1880 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_04:b3d8e19c-573a-445a-9e98-5fe67f1b3715:14499:1874531134118016:192.168.201.137@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787695712.765600:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d858c4eb000 lock: ffff9d858898de00/0x678ae32e953294fc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efd5e expref: -99 pid: 101308 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787695712.765631:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695712.765633:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695712.765636:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695712.765754:0:101919:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9d8590d43000 00010000:00010000:2.0:1787695712.765770:0:101919:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1787695712.765818:0:6059:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787695712.765826:0:6059:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695712.765964:0:6059:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031a00/0x678ae32e9532950a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efd8f expref: -99 pid: 101000 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695712.765971:0:6059:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1787695712.765989:0:6059:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:68e474e8-42cb-4207-86a3-041065654492:6059:1874531134118144:192.168.201.137@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787695712.766069:0:6059:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85c0e51480 going to sleep for 11 seconds 00010000:00010000:2.0:1787695713.491163:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695714.966785:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134118272:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695714.966945:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134118528:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695714.966981:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787695714.966987:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134118400:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695714.976779:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134118656:192.168.201.137@tcp:9:kworker.0 00000100:00100000:2.0:1787695714.976919:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134118784:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695714.976946:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134119040:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695714.976967:0:2363:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787695714.976972:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134118912:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695714.988157:0:2359:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9d85b9ce2a00 x1874531134118656/t0(0) o9->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 0 dl 1787695724 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695714.988160:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134118400:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695714.988173:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134118272:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695714.988176:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1787695714.988180:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134118656:192.168.201.137@tcp:9:kworker.0 00000100:00100000:2.0:1787695714.988183:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134118528:192.168.201.137@tcp:400:kworker.0 00000080:00200000:0.0:1787695715.552530:0:102020:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695715.552565:0:102020:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695715.552632:0:102020:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 102020 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.552644:0:102020:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 102020 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.552655:0:102020:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 102020 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695715.552684:0:102020:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0aa00 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:102020:1874531134119168:192.168.201.137@tcp:101:touch.0 00000100:00100000:0.0:1787695715.552957:0:102020:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9ecc600 going to sleep for 12 seconds 00000100:00100000:0.0:1787695715.559951:0:102020:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0aa00 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:102020:1874531134119168:192.168.201.137@tcp:101:touch.0 00010000:00010000:0.0:1787695715.559981:0:102020:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.559994:0:102020:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560007:0:102020:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560018:0:102020:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560030:0:102020:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560038:0:102020:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560046:0:102020:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695715.560057:0:102020:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9d85b9d0aa00 x1874531134119168/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 576/688 e 0 to 0 dl 1787695732 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695715.560071:0:102020:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560083:0:102020:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787695715.560089:0:102020:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787695715.560118:0:102020:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1787695715.560124:0:102020:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329518 00000080:00010000:0.0:1787695715.560129:0:102020:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695715.560133:0:102020:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329518 from it ffffb5d401eb7bf8 00010000:00010000:0.0:1787695715.560136:0:102020:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560146:0:102020:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695715.560176:0:102020:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 1 00000080:00200000:0.0:1787695715.560200:0:102020:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08),name=/ 00000002:00010000:0.0:1787695715.560213:0:102020:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787695715.560221:0:102020:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695715.560232:0:102020:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695715.560239:0:102020:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787695715.560243:0:102020:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) 00000080:00010000:0.0:1787695715.560246:0:102020:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d858ad3ec08) for lock 0x678ae32e95329518 00000080:00010000:0.0:1787695715.560250:0:102020:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329518 from it ffffb5d401eb7b38 00010000:00010000:0.0:1787695715.560254:0:102020:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695715.560261:0:102020:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787695715.560269:0:102020:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d858ad3ec08), inode mode 41ff mask 3 00000080:00200000:0.0:1787695715.560274:0:102020:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f43e.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), file ffff9d85991daf00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1787695715.560282:0:102020:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f43e.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000002:00010000:0.0:1787695715.560301:0:102020:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f43e.sanityn,[0x200000401:0x573:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1787695715.560313:0:102020:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695715.560375:0:102020:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695715.560383:0:102020:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695715.560387:0:102020:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898c800/0x678ae32e95329518 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x297ceef0832efdb9 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787695715.560423:0:102020:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 102020 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.560430:0:102020:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 102020 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787695715.560442:0:102020:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787695715.560448:0:102020:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d0aa00 x1874531134119424/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787695715.560457:0:102020:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 102020 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787695715.560477:0:102020:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0aa00 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:102020:1874531134119424:192.168.201.137@tcp:101:touch.0 00000100:00100000:0.0:1787695715.560608:0:102020:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9ecc600 going to sleep for 12 seconds 00000100:00100000:0.0:1787695715.575796:0:102020:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0aa00 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:102020:1874531134119424:192.168.201.137@tcp:101:touch.0 00010000:00010000:0.0:1787695715.575833:0:102020:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.575908:0:102020:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 4/0,1 mode: --/PR res: [0x200000401:0x573:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.575924:0:102020:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 4/0,1 mode: --/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.575936:0:102020:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.575946:0:102020:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 4/0,1 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.575959:0:102020:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.575968:0:102020:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 5/1,1 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.575976:0:102020:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787695715.575989:0:102020:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9d85b9d0aa00 x1874531134119424/t4294997980(4294997980) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695732 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787695715.576000:0:102020:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85b9d0aa00 x1874531134119424/t4294997980(4294997980) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 656/664 e 0 to 0 dl 1787695732 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787695715.576013:0:102020:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787695715.576024:0:102020:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.576033:0:102020:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 4 res 8589935617/1395 (0 0) 00000080:00200000:0.0:1787695715.576039:0:102020:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787695715.576053:0:102020:0:(namei.c:1628:ll_create_it()) VFS Op:name=f43e.sanityn, dir=[0x200000007:0x1:0x0](ffff9d858ad3ec08), intent=open|creat 00000080:00200000:0.0:1787695715.576084:0:102020:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000401:0x573:0x0](ffff9d85b3d13a88) 00000080:00200000:0.0:1787695715.576164:0:102020:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x573:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787695715.576169:0:102020:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000401:0x573:0x0](ffff9d85b3d13a88) 00000080:00010000:0.0:1787695715.576173:0:102020:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x573:0x0](ffff9d85b3d13a88) for lock 0x678ae32e9532951f 00000080:00010000:0.0:1787695715.576184:0:102020:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x573:0x0](ffff9d85b3d13a88) 00000080:00200008:0.0:1787695715.576198:0:102020:0:(file.c:1090:ll_file_open()) START file f43e.sanityn:[0x200000401:0x573:0x0](ffff9d85b3d13a88), flags 104501 00000002:00100000:0.0:1787695715.576217:0:102020:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d85b9d0aa00 x1874531134119424/t4294997980(4294997980) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 584/664 e 0 to 0 dl 1787695732 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787695715.576259:0:102020:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e9532951f from it ffff9d85990856c0 00010000:00010000:0.0:1787695715.576264:0:102020:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787695715.576273:0:102020:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695715.576295:0:102020:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000401:0x573:0x0](ffff9d85b3d13a88), inode mode 81a4 mask 40 00000080:00200008:0.0:1787695715.576334:0:102020:0:(llite_lib.c:2427:ll_setattr_raw()) START file f43e.sanityn:[0x200000401:0x573:0x0](ffff9d85b3d13a88) current size 0, valid attrs 20070, mode 9d85, uid 32210512, gid -18988, new size 18446744072543992343, atime 1787695715.000000000, mtime 1787695715.000000000, ctime 1787695715.000000000 00010000:00010000:0.0:1787695715.576366:0:102020:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787695715.576381:0:102020:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x573:0x0]: losing layout lock 00010000:00010000:0.0:1787695715.576419:0:102020:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 1/0,0 mode: --/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787695715.576426:0:102020:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787695715.576428:0:102020:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d858898de00/0x678ae32e9532951f lrc: 0/0,0 mode: --/PR res: [0x200000401:0x573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x297ceef0832efdd5 expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787695715.576448:0:102020:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d8590d43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787695715.576454:0:102020:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d85b9d09880 x1874531134119680/t0(0) o36->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695715.576478:0:102020:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09880 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:102020:1874531134119680:192.168.201.137@tcp:36:touch.0 00000100:00100000:0.0:1787695715.576550:0:102020:0:(client.c:2715:ptlrpc_set_wait()) set ffff9d85b9ecc0c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787695715.591848:0:102020:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09880 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:102020:1874531134119680:192.168.201.137@tcp:36:touch.0 00000100:00100000:2.0:1787695715.611135:0:102020:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job touch:68e474e8-42cb-4207-86a3-041065654492:102020:1874531134119808:192.168.201.137@tcp:101:touch.0 00010000:00010000:2.0:1787695715.611169:0:102020:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695715.611185:0:102020:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787695715.611199:0:102020:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9d85b9d2ca80 x1874531134119808/t0(0) o101->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 376/400 e 0 to 0 dl 1787695732 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787695715.611209:0:102020:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787695715.611222:0:102020:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695715.611234:0:102020:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8583c5c400 type 13 mode 16 res 8589935617/1395 (0 0) 00000080:00200000:2.0:1787695715.611240:0:102020:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000401:0x573:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787695715.611258:0:102020:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x573:0x0](ffff9d85b3d13a88) for lock 0x678ae32e95329526 00000080:00010000:2.0:1787695715.611279:0:102020:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x678ae32e95329526 from it ffffb5d401eb7a78 00010000:00010000:2.0:1787695715.611284:0:102020:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787695715.611296:0:102020:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695715.611318:0:102020:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695715.611334:0:102020:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 2/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787695715.611353:0:102020:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000401:0x573:0x0](000000007ce0acb1) being reconfigured ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787695715.611394:0:102020:0:(file.c:6843:ll_layout_conf()) [0x200000401:0x573:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1787695715.611399:0:102020:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787695715.611407:0:102020:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d8590d43000 lock: ffff9d8580031000/0x678ae32e95329526 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x573:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x297ceef0832efdea expref: -99 pid: 102020 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787695715.611428:0:102020:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00080000:2.0:1787695715.611465:0:102020:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3575 00000100:00080000:2.0:1787695715.611475:0:102020:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695715.611485:0:102020:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 3407872, lost: 0. 00000100:00100000:2.0:1787695715.611508:0:102020:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134119936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787695715.611655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134119936:192.168.201.137@tcp:8:touch.0 00000100:00100000:0.0:1787695715.622352:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d8590d43000: committing for last_committed 4294969647 gen 17 00000100:00080000:0.0:1787695715.622371:0:2359:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff9d8590d43000: connect to target with instance 1 00000100:00080000:0.0:1787695715.622381:0:2359:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff9d8590d43000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1787695715.622394:0:2359:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1787695715.622398:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1787695715.622409:0:2359:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1787695715.622414:0:2359:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1787695715.622430:0:2359:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000c49ee780): req@ffff9d85babd5880 x1874531134120064/t0(0) o2->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787695715.622460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134119936:192.168.201.137@tcp:8:touch.0 00010000:00010000:2.0:1787695717.587018:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695719.635059:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695720.083066:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2363:1874531134120192:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695720.083172:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134120448:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695720.083227:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134120704:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695720.083303:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134120960:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695723.731040:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695725.203111:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134121216:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695725.203232:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134121472:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695725.203272:0:2363:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134121728:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695725.779033:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695729.875002:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787695730.323036:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134122112:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695730.323138:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134122368:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695730.323182:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134122624:192.168.201.137@tcp:400:kworker.0 00010000:00010000:0.0:1787695730.898993:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787695731.346942:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695715/real 1787695715] req@ffff9d85babd5880 x1874531134120064/t0(0) o2->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 440/432 e 0 to 1 dl 1787695731 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1787695731.353403:0:2362:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787695731.353410:0:2362:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff9d8590d43000: Connection to lustre-OST0001 (at 192.168.201.137@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1787695731.356902:0:2362:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1787695731.357023:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134122880:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695731.357129:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134123136:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695731.357164:0:2362:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134123520:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695731.357190:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9ce1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134123264:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787695731.923006:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695736.019048:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787695736.466964:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695720/real 1787695720] req@ffff9d85b9ce1f80 x1874531134120576/t0(0) o400->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695736.474680:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce1f80 x1874531134120576/t0(0) o400->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695736 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695736.474691:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c484800 already not connected (conn 18, was 17): CONNECTING 00000100:00000400:2.0:1787695736.474697:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695720/real 1787695720] req@ffff9d85b9ce0a80 x1874531134120832/t0(0) o400->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695736.474707:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce0a80 x1874531134120832/t0(0) o400->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695736 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695736.474713:0:2362:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787695736.474717:0:2362:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff9d858c4eb000: Connection to lustre-OST0000 (at 192.168.201.137@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787695736.482220:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695736.482229:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695736.483670:0:2362:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1787695736.483685:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695736.483692:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787695736.483696:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134122880:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695736.483698:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787695736.483708:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2362:1874531134121088:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695736.483720:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3489 00000100:00100000:2.0:1787695736.483720:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134120576:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695736.483737:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695736.483746:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134120832:192.168.201.137@tcp:400:kworker.0 00000008:00100000:0.0:1787695736.483749:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695736.483779:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134124544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695736.483805:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695736.483809:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695736.483811:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695736.483815:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3489 00000100:00080000:0.0:1787695736.483822:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695736.483828:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695736.483844:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134124672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695736.483890:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695736.483895:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787695736.483897:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695736.483903:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 221 00000100:00080000:0.0:1787695736.483912:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787695736.483932:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134124800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695736.483945:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695736.483947:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695736.483950:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695736.483954:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3591 00000100:00080000:0.0:1787695736.483961:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695736.483968:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695736.484004:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134124928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695736.484017:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695736.484019:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1787695737.042990:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787695737.490978:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695720/real 1787695720] req@ffff9d85b9ce0700 x1874531134120320/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695737.491010:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce0700 x1874531134120320/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695737 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695737.491035:0:2362:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787695737.491039:0:2362:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9d8590d43000: Connection to lustre-MDT0000 (at 192.168.201.137@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787695737.498152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.498157:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695737.506621:0:2362:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1787695737.506632:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506639:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695737.506642:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695737.506652:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 221 00000100:00100000:2.0:1787695737.506653:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134120320:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695737.506671:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695737.506682:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695737.506717:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134125056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695737.506736:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506739:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506741:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506743:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506745:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506747:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695737.506749:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695737.506752:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 278 00000100:00080000:0.0:1787695737.506757:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695737.506760:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695737.506773:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134125184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695737.506781:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506783:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506785:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506787:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506793:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506795:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506796:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506798:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506800:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506802:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506804:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506806:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695737.506807:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695737.506809:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1787695738.067042:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787695741.586971:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695725/real 1787695725] req@ffff9d85babd7100 x1874531134121344/t0(0) o400->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695741.587043:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85babd7100 x1874531134121344/t0(0) o400->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695741 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695741.587064:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c485000 already not connected (conn 22, was 21): CONNECTING 00000100:00000400:2.0:1787695741.587073:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695725/real 1787695725] req@ffff9d85babd7b80 x1874531134121856/t0(0) o400->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695741.587098:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85babd7b80 x1874531134121856/t0(0) o400->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695741 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695741.587107:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c4ed000 already not connected (conn 12, was 11): CONNECTING 00000100:00080000:0.0:1787695741.587110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695741.587120:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787695741.587124:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134121344:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695741.587125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695741.587133:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787695741.587137:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134121856:192.168.201.137@tcp:400:kworker.0 00000100:00080000:0.0:1787695741.587138:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695741.587146:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695741.587149:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695741.587153:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695741.587157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695741.587165:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3638 00000100:00080000:0.0:1787695741.587177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787695741.587185:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695741.587195:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695741.587227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134125312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695741.587253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695741.587255:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695741.587258:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695741.587260:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695741.587261:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695741.587263:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1787695742.162980:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787695742.610944:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695742.610953:0:89493:0:(pinger.c:247:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695742.610958:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000400:2.0:1787695742.610961:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695725/real 1787695725] req@ffff9d85babd4700 x1874531134121600/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787695742.610962:0:89493:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695742.610968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695742.610978:0:89493:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695742.610983:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695742.610992:0:89493:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695742.610999:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695742.611003:0:89493:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695742.611004:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695742.611006:0:89493:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695742.611009:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695742.611012:0:89493:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787695742.611018:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85babd4700 x1874531134121600/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695742 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695742.611028:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9d858c4ef800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1787695742.611044:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134121600:192.168.201.137@tcp:400:kworker.0 00010000:00010000:0.0:1787695743.187021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695744.211079:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1787695745.683007:0:100989:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x573:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787695745.683030:0:100989:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787695745.683077:0:100989:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x573:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00080000:2.0:1787695746.642986:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695746.643000:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695746.643005:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695746.643022:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695746.643024:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695746.643027:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695746.643030:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695746.643032:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695746.643035:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695746.643037:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695746.643039:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695746.643042:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695746.643044:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695746.643047:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:2.0:1787695746.706965:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695730/real 1787695730] req@ffff9d85b9ce1c00 x1874531134122240/t0(0) o400->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787695746.707257:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707266:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787695746.707271:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695746.707282:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3643 00000100:00080000:0.0:1787695746.707295:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787695746.707309:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787695746.707349:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134125440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695746.707378:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707383:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695746.707387:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707390:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695746.707393:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695746.707399:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3643 00000100:00080000:0.0:1787695746.707406:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787695746.707411:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695746.707425:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695746.707446:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134125568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695746.707456:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707460:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695746.707463:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707467:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695746.707471:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707475:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695746.707477:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695746.707482:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3643 00000100:00080000:0.0:1787695746.707487:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787695746.707491:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695746.707495:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695746.707509:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134125696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695746.707545:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707548:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695746.707550:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695746.707555:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3643 00000100:00080000:0.0:1787695746.707562:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787695746.707568:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695746.707578:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695746.707599:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134125824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695746.707627:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707641:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695746.707645:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707648:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695746.707651:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695746.707654:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787695746.726227:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce1c00 x1874531134122240/t0(0) o400->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695746 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695746.726239:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c485000 already not connected (conn 23, was 21): CONNECTING 00000100:00000400:2.0:1787695746.726248:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695730/real 1787695730] req@ffff9d85b9ce1500 x1874531134122752/t0(0) o400->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695746.726262:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce1500 x1874531134122752/t0(0) o400->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695746 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695746.726269:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c4ed000 already not connected (conn 13, was 11): CONNECTING 00000100:00100000:2.0:1787695746.726282:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134122240:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695746.726287:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134122752:192.168.201.137@tcp:400:kworker.0 00000100:00000400:2.0:1787695747.731005:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695731/real 1787695731] req@ffff9d85c0fb3b80 x1874531134123648/t0(0) o400->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695747.731053:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85c0fb3b80 x1874531134123648/t0(0) o400->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695747.731067:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c4ed000 already not connected (conn 13, was 11): CONNECTING 00000100:00000400:2.0:1787695747.731083:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695730/real 1787695730] req@ffff9d85b9ce3b80 x1874531134122496/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695747.731162:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce3b80 x1874531134122496/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695747.731177:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9d858c4ef800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1787695747.731198:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134123648:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695747.731225:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134122496:192.168.201.137@tcp:400:kworker.0 00000100:00000400:2.0:1787695747.731306:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695731/real 1787695731] req@ffff9d85c0fb1c00 x1874531134123520/t0(0) o400->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695747.731341:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85c0fb1c00 x1874531134123520/t0(0) o400->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695747.731349:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c4ec000 already not connected (conn 24, was 22): CONNECTING 00000100:00000400:2.0:1787695747.731355:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695731/real 1787695731] req@ffff9d85b9ce2a00 x1874531134123136/t0(0) o400->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695747.731372:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce2a00 x1874531134123136/t0(0) o400->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695747.731394:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9d858c485000 already not connected (conn 23, was 21): CONNECTING 00000100:00080000:0.0:1787695747.731405:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000400:2.0:1787695747.731410:0:2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695730/real 1787695730] req@ffff9d85b9ce3800 x1874531134122112/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787695747.731422:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731426:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731431:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695747.731435:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695747.731447:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3644 00000100:00080000:0.0:1787695747.731476:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 5s 00000100:00100000:2.0:1787695747.731488:0:2362:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce3800 x1874531134122112/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695747 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787695747.731494:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:2.0:1787695747.731521:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9d858c487000 already not connected (conn 3, was 1): CONNECTING 00000008:00100000:0.0:1787695747.731523:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695747.731547:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2362:1874531134123520:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695747.731569:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134123136:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695747.731577:0:2362:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:68e474e8-42cb-4207-86a3-041065654492:2362:1874531134122112:192.168.201.137@tcp:400:kworker.0 00000100:00100000:0.0:1787695747.731585:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134125952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695747.731635:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731640:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731643:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731648:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731651:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731656:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695747.731659:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695747.731666:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3644 00000100:00080000:0.0:1787695747.731672:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787695747.731679:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695747.731685:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695747.731699:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134126080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695747.731713:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731717:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731721:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731724:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731735:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731739:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731742:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731746:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731748:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731752:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731754:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731757:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695747.731759:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695747.731762:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1787695748.307023:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787695748.754998:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695731/real 1787695731] req@ffff9d85b9ce3100 x1874531134123008/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695748.755033:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85b9ce3100 x1874531134123008/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695748 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695748.755040:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9d858c487000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:2.0:1787695748.755045:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695731/real 1787695731] req@ffff9d85c0fb3800 x1874531134123392/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695748.755054:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85c0fb3800 x1874531134123392/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695748 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695748.755058:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9d858c4ef800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1787695748.755071:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9ce3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134123008:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695748.755075:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134123392:192.168.201.137@tcp:400:kworker.0 00010000:00010000:0.0:1787695749.331049:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695750.354999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787695753.874977:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695736/real 1787695736] req@ffff9d85c0fb2d80 x1874531134123904/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695753.875008:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85c0fb2d80 x1874531134123904/t0(0) o400->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695753 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695753.875016:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9d858c487000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:2.0:1787695753.875021:0:2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787695736/real 1787695736] req@ffff9d85c0fb0a80 x1874531134124160/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695753.875030:0:2363:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d85c0fb0a80 x1874531134124160/t0(0) o400->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1787695753 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695753.875035:0:2363:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9d858c4ef800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1787695753.875047:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:68e474e8-42cb-4207-86a3-041065654492:2363:1874531134123904:192.168.201.137@tcp:400:kworker.0 00000100:00100000:2.0:1787695753.875052:0:2363:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2363:1874531134124160:192.168.201.137@tcp:400:kworker.0 00010000:00010000:2.0:1787695754.451046:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695755.475125:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695756.498981:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1787695757.586957:0:2351:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.201.137@tcp (192.168.201.137:988) timed out; the network or node may be down. 00010000:00010000:2.0:1787695760.595022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695761.619023:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695762.642991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787695764.115084:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.137:988 00000400:00000100:0.0:1787695764.115122:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787695764.115135:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695764.115177:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695764.115186:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695764.115194:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695764.115201:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695764.115215:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00080000:2.0:1787695766.098970:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695766.098981:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695766.098986:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695766.098995:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3667 00000100:00080000:2.0:1787695766.099013:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787695766.099020:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695766.099058:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134127104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695766.099085:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695766.099088:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695766.099090:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695766.099094:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3668 00000100:00080000:2.0:1787695766.099098:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787695766.099102:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695766.099111:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695766.099124:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134127232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695766.099132:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695766.099134:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695766.099136:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695766.099139:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3667 00000100:00080000:2.0:1787695766.099141:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787695766.099144:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695766.099147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695766.099156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134127360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695766.099162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695766.099164:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787695766.099166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695766.099169:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695766.099170:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695766.099172:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3668 00000100:00080000:2.0:1787695766.099175:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787695766.099177:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695766.099180:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695766.099186:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134127488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695766.099193:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695766.099196:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695766.099197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695766.099199:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3667 00000100:00080000:2.0:1787695766.099201:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787695766.099203:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695766.099205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695766.099212:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134127616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695766.099218:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695766.099220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695766.099222:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695766.099225:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3667 00000100:00080000:2.0:1787695766.099228:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787695766.099230:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695766.099234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695766.099245:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134127744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695766.739027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695767.763055:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695768.787022:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787695771.282957:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695771.282965:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787695771.282969:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695771.282977:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3672 00000100:00080000:0.0:1787695771.282986:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695771.282992:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787695771.283020:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134127872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695771.283041:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695771.283049:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695771.283050:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695771.283054:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3672 00000100:00080000:0.0:1787695771.283057:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695771.283060:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695771.283068:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695771.283080:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134128000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695771.283092:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695771.283100:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695771.283105:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695771.283110:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3672 00000100:00080000:0.0:1787695771.283114:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695771.283118:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695771.283125:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695771.283141:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134128128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695771.283152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695771.283155:0:100989:0:(pinger.c:247:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787695771.283158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695771.283162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695771.283164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695771.283168:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3672 00000100:00080000:0.0:1787695771.283172:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695771.283177:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695771.283183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695771.283197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134128256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695771.283213:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695771.283218:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695771.283221:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695771.283227:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3672 00000100:00080000:0.0:1787695771.283232:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695771.283236:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695771.283242:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695771.283257:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134128384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695771.283270:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695771.283273:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695771.283275:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695771.283280:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3672 00000100:00080000:0.0:1787695771.283285:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695771.283290:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695771.283296:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695771.283317:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134128512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695772.883041:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695773.907005:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787695774.355179:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.137:988 00000400:00000100:2.0:1787695774.355229:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787695774.355244:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695774.355273:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695774.355279:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695774.355299:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695774.355308:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695774.355319:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787695774.931020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787695776.402960:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695776.402966:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787695776.402969:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695776.402979:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3678 00000100:00080000:0.0:1787695776.402990:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695776.402997:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787695776.403027:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134128640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695776.403042:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695776.403045:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695776.403047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695776.403050:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3678 00000100:00080000:0.0:1787695776.403055:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695776.403062:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695776.403071:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695776.403082:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134128768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695776.403090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695776.403092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695776.403094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695776.403097:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3678 00000100:00080000:0.0:1787695776.403110:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695776.403116:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695776.403122:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695776.403140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134128896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695776.403155:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695776.403159:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695776.403162:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695776.403180:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3663 00000100:00080000:0.0:1787695776.403187:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787695776.403195:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695776.403200:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695776.403218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134129024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695776.403238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695776.403242:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695776.403246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695776.403256:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3678 00000100:00080000:0.0:1787695776.403260:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695776.403265:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695776.403270:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695776.403284:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134129152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695776.403311:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695776.403315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695776.403317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695776.403322:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3678 00000100:00080000:0.0:1787695776.403331:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695776.403336:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695776.403343:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695776.403373:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134129280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695776.403393:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695776.403403:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695776.403407:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695776.403417:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3678 00000100:00080000:0.0:1787695776.403423:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695776.403428:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695776.403434:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695776.403449:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134129408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695779.026994:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695780.051031:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695781.075017:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787695781.522984:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695781.522992:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787695781.522996:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695781.523005:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3683 00000100:00080000:0.0:1787695781.523015:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695781.523023:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787695781.523058:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134129536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695781.523088:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695781.523093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695781.523095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695781.523100:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3683 00000100:00080000:0.0:1787695781.523104:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695781.523110:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695781.523120:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695781.523138:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134129664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695781.523155:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695781.523159:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695781.523161:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695781.523165:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3683 00000100:00080000:0.0:1787695781.523170:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695781.523175:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695781.523181:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695781.523196:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134129792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695781.523237:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695781.523241:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695781.523243:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695781.523248:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3683 00000100:00080000:0.0:1787695781.523253:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695781.523258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695781.523265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695781.523286:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134129920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695781.523300:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695781.523305:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695781.523308:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695781.523318:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3683 00000100:00080000:0.0:1787695781.523341:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695781.523351:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695781.523359:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695781.523377:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134130048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695781.523392:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695781.523396:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695781.523398:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695781.523405:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3683 00000100:00080000:0.0:1787695781.523411:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695781.523419:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695781.523426:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695781.523441:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134130176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695781.523455:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695781.523458:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695781.523461:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695781.523466:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3683 00000100:00080000:0.0:1787695781.523470:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695781.523475:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695781.523481:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695781.523496:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134130304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:2.0:1787695783.571138:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.137:988 00000400:00000100:2.0:1787695783.571209:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787695783.571226:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695783.571267:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695783.571279:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695783.571289:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695783.571298:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695783.571309:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787695783.571320:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787695785.171014:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695786.195033:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695786.642947:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695786.642956:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695786.642959:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695786.642967:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3688 00000100:00080000:2.0:1787695786.642976:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695786.642982:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695786.643012:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134130432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695786.643036:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695786.643039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695786.643040:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695786.643043:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3688 00000100:00080000:2.0:1787695786.643046:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695786.643049:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695786.643056:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695786.643069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134130560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695786.643081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695786.643084:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695786.643084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695786.643087:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3688 00000100:00080000:2.0:1787695786.643090:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695786.643092:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695786.643095:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695786.643104:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134130688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695786.643134:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695786.643136:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695786.643137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695786.643140:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3688 00000100:00080000:2.0:1787695786.643144:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695786.643147:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695786.643151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695786.643162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134130816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695786.643170:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695786.643179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695786.643181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695786.643186:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3688 00000100:00080000:2.0:1787695786.643189:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695786.643192:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695786.643196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695786.643206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134130944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695786.643227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695786.643230:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695786.643231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695786.643236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3688 00000100:00080000:2.0:1787695786.643242:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695786.643248:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695786.643253:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695786.643262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134131072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695786.643273:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695786.643276:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695786.643277:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695786.643282:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3688 00000100:00080000:2.0:1787695786.643286:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695786.643291:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695786.643297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695786.643310:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134131200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695787.219019:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695791.315109:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695792.338987:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695793.363110:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787695796.882981:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695796.882989:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787695796.882993:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695796.883001:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3698 00000100:00080000:0.0:1787695796.883010:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695796.883016:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787695796.883049:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134132224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695796.883085:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695796.883088:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695796.883090:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695796.883093:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3698 00000100:00080000:0.0:1787695796.883097:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695796.883101:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695796.883109:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695796.883123:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134132352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695796.883132:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695796.883136:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695796.883137:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695796.883140:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3698 00000100:00080000:0.0:1787695796.883143:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695796.883145:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695796.883149:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695796.883158:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134132480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695796.883164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695796.883168:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695796.883169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695796.883171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3698 00000100:00080000:0.0:1787695796.883174:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695796.883176:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695796.883193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695796.883203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134132608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695796.883209:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695796.883212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695796.883213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695796.883216:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3698 00000100:00080000:0.0:1787695796.883220:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695796.883225:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695796.883231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695796.883242:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134132736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695796.883249:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695796.883255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695796.883259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695796.883268:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3698 00000100:00080000:0.0:1787695796.883275:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695796.883281:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695796.883290:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695796.883312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134132864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695796.883326:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695796.883333:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695796.883335:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695796.883340:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3698 00000100:00080000:0.0:1787695796.883344:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695796.883349:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695796.883354:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695796.883370:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134132992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695797.459049:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695798.483084:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695799.506997:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695803.603026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695804.627025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695805.651102:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695809.747024:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695810.771029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695811.795024:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787695812.242938:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695812.242947:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787695812.242951:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695812.242961:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3714 00000100:00080000:0.0:1787695812.242973:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695812.242981:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787695812.243015:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134134912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695812.243037:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695812.243039:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695812.243041:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695812.243044:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3714 00000100:00080000:0.0:1787695812.243047:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695812.243052:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695812.243063:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695812.243080:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134135040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695812.243095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695812.243098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695812.243100:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695812.243113:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3714 00000100:00080000:0.0:1787695812.243121:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695812.243124:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695812.243129:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695812.243140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134135168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695812.243147:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695812.243150:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695812.243151:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695812.243154:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3714 00000100:00080000:0.0:1787695812.243156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695812.243159:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695812.243163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695812.243172:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134135296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695812.243182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695812.243185:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787695812.243187:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695812.243196:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3714 00000100:00080000:0.0:1787695812.243201:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695812.243209:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695812.243215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695812.243230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134135424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695812.243254:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695812.243258:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787695812.243261:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695812.243272:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3714 00000100:00080000:0.0:1787695812.243281:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695812.243286:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695812.243292:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695812.243311:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134135552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787695812.243331:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787695812.243336:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787695812.243338:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787695812.243359:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3714 00000100:00080000:0.0:1787695812.243366:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787695812.243369:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787695812.243375:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787695812.243385:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134135680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695815.891031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695816.914993:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695817.362936:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695817.362947:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695817.362951:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695817.362962:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3719 00000100:00080000:2.0:1787695817.362984:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695817.362992:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695817.363030:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134135808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695817.363056:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695817.363060:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695817.363062:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695817.363068:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3719 00000100:00080000:2.0:1787695817.363072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695817.363076:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695817.363095:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695817.363113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134135936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695817.363127:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695817.363130:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695817.363132:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695817.363137:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3719 00000100:00080000:2.0:1787695817.363141:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695817.363146:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695817.363152:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695817.363166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134136064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695817.363177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695817.363181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695817.363183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695817.363192:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3719 00000100:00080000:2.0:1787695817.363198:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695817.363203:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695817.363209:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695817.363224:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134136192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695817.363238:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695817.363248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695817.363252:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695817.363260:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3719 00000100:00080000:2.0:1787695817.363264:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695817.363273:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695817.363285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695817.363301:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134136320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695817.363320:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695817.363323:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695817.363325:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695817.363331:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3719 00000100:00080000:2.0:1787695817.363335:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695817.363344:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695817.363351:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695817.363366:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134136448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695817.363384:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695817.363390:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695817.363409:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695817.363420:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3719 00000100:00080000:2.0:1787695817.363426:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695817.363435:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695817.363444:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695817.363458:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134136576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695817.939022:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695822.035042:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695823.059007:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695824.083008:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695828.179059:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695829.203084:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695830.227003:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695834.323057:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695835.347008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695836.371014:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695840.467950:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695841.491010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695842.963005:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695842.963016:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695842.963020:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695842.963031:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3744 00000100:00080000:2.0:1787695842.963043:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695842.963052:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695842.963100:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134140288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695842.963126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695842.963130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695842.963131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695842.963134:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3744 00000100:00080000:2.0:1787695842.963136:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695842.963139:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695842.963164:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695842.963176:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134140416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695842.963184:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695842.963189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695842.963191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695842.963200:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3744 00000100:00080000:2.0:1787695842.963203:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695842.963206:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695842.963210:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695842.963219:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134140544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695842.963226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695842.963228:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695842.963231:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695842.963234:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3744 00000100:00080000:2.0:1787695842.963236:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695842.963247:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695842.963250:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695842.963259:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134140672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695842.963265:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695842.963268:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695842.963269:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695842.963274:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3744 00000100:00080000:2.0:1787695842.963280:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695842.963283:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695842.963287:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695842.963295:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134140800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695842.963302:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695842.963305:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695842.963306:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695842.963309:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3744 00000100:00080000:2.0:1787695842.963311:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695842.963313:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695842.963317:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695842.963325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134140928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695842.963339:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695842.963347:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695842.963351:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695842.963355:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3744 00000100:00080000:2.0:1787695842.963357:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695842.963363:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695842.963366:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695842.963378:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134141056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695847.634977:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695848.082959:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695848.082967:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695848.082970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695848.082978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3749 00000100:00080000:2.0:1787695848.082988:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695848.082994:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695848.083028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134141184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695848.083050:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695848.083066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695848.083068:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695848.083074:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3749 00000100:00080000:2.0:1787695848.083079:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695848.083084:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695848.083094:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695848.083115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134141312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695848.083128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695848.083131:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695848.083133:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695848.083136:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3749 00000100:00080000:2.0:1787695848.083147:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695848.083150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695848.083154:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695848.083164:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134141440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695848.083171:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695848.083173:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695848.083175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695848.083178:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3749 00000100:00080000:2.0:1787695848.083182:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695848.083186:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695848.083192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695848.083208:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134141568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695848.083238:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695848.083241:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695848.083243:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695848.083247:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3749 00000100:00080000:2.0:1787695848.083251:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695848.083257:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695848.083265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695848.083279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134141696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695848.083286:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695848.083289:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695848.083290:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695848.083294:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3749 00000100:00080000:2.0:1787695848.083296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695848.083301:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695848.083306:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695848.083317:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134141824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695848.083332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695848.083334:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695848.083336:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695848.083342:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3749 00000100:00080000:2.0:1787695848.083350:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695848.083355:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695848.083359:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695848.083369:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134141952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:0.0:1787695851.155109:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787695851.155154:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787695851.155171:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695851.155208:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695851.155220:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695851.155231:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695851.155243:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695851.155255:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695851.155267:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787695853.779034:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695858.322937:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695858.322947:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695858.322950:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695858.322959:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3760 00000100:00080000:2.0:1787695858.322968:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695858.322974:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695858.323006:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134142976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695858.323028:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695858.323031:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695858.323032:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695858.323035:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3760 00000100:00080000:2.0:1787695858.323038:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695858.323040:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695858.323048:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695858.323056:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134143104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695858.323064:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695858.323069:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695858.323071:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695858.323079:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3760 00000100:00080000:2.0:1787695858.323082:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695858.323084:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695858.323087:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695858.323096:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134143232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695858.323102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695858.323104:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695858.323106:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695858.323108:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3760 00000100:00080000:2.0:1787695858.323119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695858.323123:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695858.323126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695858.323135:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134143360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695858.323143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695858.323146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695858.323147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695858.323150:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3760 00000100:00080000:2.0:1787695858.323152:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695858.323155:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695858.323158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695858.323164:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134143488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695858.323169:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695858.323171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695858.323173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695858.323176:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3760 00000100:00080000:2.0:1787695858.323178:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695858.323184:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695858.323187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695858.323198:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134143616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695858.323204:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695858.323208:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695858.323212:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695858.323220:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3760 00000100:00080000:2.0:1787695858.323225:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695858.323230:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695858.323235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695858.323245:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134143744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695858.899010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695859.923021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695863.442937:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695863.442945:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695863.442949:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695863.442956:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3765 00000100:00080000:2.0:1787695863.442966:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695863.442972:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695863.443013:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134143872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695863.443033:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695863.443035:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695863.443037:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695863.443040:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3765 00000100:00080000:2.0:1787695863.443042:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695863.443045:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695863.443053:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695863.443062:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134144000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695863.443071:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695863.443076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695863.443079:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695863.443084:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3765 00000100:00080000:2.0:1787695863.443086:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695863.443089:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695863.443093:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695863.443102:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134144128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695863.443110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695863.443112:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695863.443114:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695863.443116:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3765 00000100:00080000:2.0:1787695863.443125:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695863.443128:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695863.443131:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695863.443141:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134144256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695863.443149:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695863.443152:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695863.443153:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695863.443156:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3765 00000100:00080000:2.0:1787695863.443158:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695863.443161:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695863.443164:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695863.443173:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134144384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695863.443189:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695863.443195:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695863.443197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695863.443202:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3765 00000100:00080000:2.0:1787695863.443205:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695863.443211:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695863.443218:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695863.443227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134144512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695863.443243:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695863.443247:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695863.443249:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695863.443255:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3765 00000100:00080000:2.0:1787695863.443258:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695863.443262:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695863.443286:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695863.443303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134144640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695865.042970:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695866.067009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695868.562931:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695868.562942:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695868.562946:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695868.562956:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3770 00000100:00080000:2.0:1787695868.562965:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695868.562971:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695868.563005:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134144768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695868.563029:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695868.563033:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695868.563036:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695868.563041:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3770 00000100:00080000:2.0:1787695868.563046:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695868.563051:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695868.563062:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695868.563083:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134144896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695868.563095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695868.563099:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695868.563101:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695868.563106:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3770 00000100:00080000:2.0:1787695868.563110:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695868.563116:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695868.563121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695868.563136:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134145024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695868.563145:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695868.563147:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695868.563148:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695868.563151:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3770 00000100:00080000:2.0:1787695868.563156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695868.563161:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695868.563166:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695868.563190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134145152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695868.563214:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695868.563223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695868.563228:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695868.563237:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3770 00000100:00080000:2.0:1787695868.563246:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695868.563252:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695868.563259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695868.563273:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134145280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695868.563281:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695868.563287:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695868.563289:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695868.563292:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3770 00000100:00080000:2.0:1787695868.563295:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695868.563298:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695868.563302:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695868.563313:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134145408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695868.563325:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695868.563328:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695868.563329:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695868.563331:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3770 00000100:00080000:2.0:1787695868.563334:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695868.563336:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695868.563339:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695868.563353:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134145536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695871.187073:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695872.210997:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695877.331005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695878.355077:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695883.475027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695883.923007:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695883.923020:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695883.923025:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695883.923036:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3785 00000100:00080000:2.0:1787695883.923048:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695883.923056:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695883.923100:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134147456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695883.923130:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695883.923133:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695883.923136:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695883.923141:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3785 00000100:00080000:2.0:1787695883.923145:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695883.923150:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695883.923162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695883.923179:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134147584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695883.923196:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695883.923206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695883.923212:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695883.923217:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3785 00000100:00080000:2.0:1787695883.923219:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695883.923223:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695883.923229:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695883.923243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134147712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695883.923250:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695883.923253:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695883.923254:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695883.923257:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3785 00000100:00080000:2.0:1787695883.923268:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695883.923271:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695883.923275:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695883.923291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134147840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695883.923299:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695883.923302:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695883.923304:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695883.923308:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3785 00000100:00080000:2.0:1787695883.923312:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695883.923316:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695883.923319:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695883.923328:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134147968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695883.923335:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695883.923337:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695883.923338:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695883.923341:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3785 00000100:00080000:2.0:1787695883.923344:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695883.923348:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695883.923351:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695883.923362:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134148096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695883.923434:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695883.923450:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695883.923452:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695883.923458:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3785 00000100:00080000:2.0:1787695883.923463:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695883.923469:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695883.923476:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695883.923499:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134148224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695884.499017:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695889.619021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695890.643091:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695894.162953:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695894.162965:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695894.162970:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695894.162980:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3795 00000100:00080000:2.0:1787695894.162992:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695894.163000:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695894.163039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134149248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695894.163067:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695894.163070:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695894.163072:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695894.163075:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3795 00000100:00080000:2.0:1787695894.163082:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695894.163087:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695894.163097:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695894.163113:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134149376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695894.163135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695894.163138:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695894.163141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695894.163146:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3795 00000100:00080000:2.0:1787695894.163151:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695894.163156:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695894.163163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695894.163177:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134149504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695894.163195:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695894.163198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695894.163200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695894.163204:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3795 00000100:00080000:2.0:1787695894.163208:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695894.163211:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695894.163215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695894.163223:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134149632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695894.163235:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695894.163239:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695894.163241:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695894.163249:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3795 00000100:00080000:2.0:1787695894.163254:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695894.163272:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695894.163278:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695894.163295:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134149760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695894.163311:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695894.163318:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695894.163320:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695894.163325:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3795 00000100:00080000:2.0:1787695894.163332:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695894.163341:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695894.163350:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695894.163365:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134149888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695894.163382:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695894.163387:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695894.163389:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695894.163394:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3795 00000100:00080000:2.0:1787695894.163400:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695894.163406:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695894.163415:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695894.163433:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134150016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695895.763025:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695896.787011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695901.907018:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695902.931026:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695904.402975:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695904.402990:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695904.402994:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695904.403004:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3806 00000100:00080000:2.0:1787695904.403015:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695904.403021:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695904.403052:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134151040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695904.403074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695904.403077:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695904.403078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695904.403116:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3806 00000100:00080000:2.0:1787695904.403121:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695904.403127:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695904.403137:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695904.403154:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134151168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695904.403167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695904.403171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695904.403173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695904.403177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3806 00000100:00080000:2.0:1787695904.403183:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695904.403188:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695904.403194:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695904.403208:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134151296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695904.403225:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695904.403229:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695904.403231:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695904.403237:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3806 00000100:00080000:2.0:1787695904.403241:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695904.403245:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695904.403249:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695904.403261:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134151424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695904.403294:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695904.403297:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695904.403298:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695904.403303:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3806 00000100:00080000:2.0:1787695904.403308:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695904.403314:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695904.403320:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695904.403330:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134151552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695904.403349:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695904.403352:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695904.403353:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695904.403356:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3806 00000100:00080000:2.0:1787695904.403358:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695904.403360:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695904.403363:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695904.403369:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134151680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695904.403381:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695904.403385:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695904.403387:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695904.403390:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3806 00000100:00080000:2.0:1787695904.403396:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695904.403399:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695904.403405:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695904.403415:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134151808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695908.050995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695909.074999:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695909.522978:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695909.522989:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695909.523009:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695909.523020:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3811 00000100:00080000:2.0:1787695909.523031:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695909.523039:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695909.523087:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134151936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695909.523122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695909.523126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695909.523128:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695909.523133:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3811 00000100:00080000:2.0:1787695909.523136:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695909.523140:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695909.523150:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695909.523163:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134152064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695909.523174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695909.523179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695909.523183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695909.523190:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3811 00000100:00080000:2.0:1787695909.523194:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695909.523198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695909.523204:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695909.523219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134152192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695909.523230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695909.523234:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695909.523236:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695909.523245:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3811 00000100:00080000:2.0:1787695909.523252:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695909.523257:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695909.523262:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695909.523276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134152320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695909.523295:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695909.523300:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695909.523303:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695909.523311:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3811 00000100:00080000:2.0:1787695909.523319:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695909.523328:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695909.523338:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695909.523356:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134152448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695909.523374:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695909.523381:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695909.523385:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695909.523391:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3811 00000100:00080000:2.0:1787695909.523397:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695909.523407:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695909.523416:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695909.523437:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134152576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695909.523451:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695909.523458:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695909.523462:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695909.523470:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3811 00000100:00080000:2.0:1787695909.523480:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695909.523490:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695909.523499:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695909.523519:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134152704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787695914.195069:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695914.642975:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695914.642989:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695914.642995:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695914.643009:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3816 00000100:00080000:2.0:1787695914.643023:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695914.643034:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695914.643084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134152832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695914.643114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695914.643118:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695914.643120:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695914.643127:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3816 00000100:00080000:2.0:1787695914.643131:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695914.643136:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695914.643148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695914.643163:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134152960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695914.643179:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695914.643187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695914.643191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695914.643205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3816 00000100:00080000:2.0:1787695914.643210:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695914.643215:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695914.643221:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695914.643235:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134153088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695914.643246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695914.643251:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695914.643254:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695914.643258:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3816 00000100:00080000:2.0:1787695914.643262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695914.643267:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695914.643272:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695914.643284:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134153216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695914.643296:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695914.643300:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695914.643302:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695914.643307:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3816 00000100:00080000:2.0:1787695914.643312:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695914.643317:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695914.643322:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695914.643338:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134153344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695914.643352:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695914.643357:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695914.643359:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695914.643364:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3816 00000100:00080000:2.0:1787695914.643378:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695914.643384:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695914.643391:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695914.643412:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134153472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695914.643424:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695914.643430:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695914.643435:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695914.643443:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3816 00000100:00080000:2.0:1787695914.643448:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695914.643453:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695914.643458:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695914.643475:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134153600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787695915.219047:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787695917.715107:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787695917.715145:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787695917.715158:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695917.715189:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695917.715196:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695917.715202:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695917.715208:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695917.715217:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695917.715231:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:0.0:1787695921.363075:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695927.506991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695933.651055:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695939.795074:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695945.939051:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695952.083019:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695958.227023:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695964.371056:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695969.491074:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695970.515121:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695975.635048:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787695976.082993:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695976.083004:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695976.083008:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695976.083019:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3877 00000100:00080000:2.0:1787695976.083030:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695976.083038:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787695976.083079:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134163584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695976.083112:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695976.083116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695976.083119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695976.083124:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3877 00000100:00080000:2.0:1787695976.083128:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695976.083132:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695976.083142:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695976.083156:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134163712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695976.083175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695976.083179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695976.083181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695976.083185:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3877 00000100:00080000:2.0:1787695976.083190:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695976.083198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695976.083206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695976.083220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134163840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695976.083231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695976.083236:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695976.083237:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695976.083241:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3877 00000100:00080000:2.0:1787695976.083246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695976.083250:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695976.083256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695976.083272:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134163968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695976.083283:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695976.083287:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695976.083289:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695976.083293:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3877 00000100:00080000:2.0:1787695976.083298:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695976.083304:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695976.083311:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695976.083332:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134164096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695976.083367:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695976.083375:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695976.083380:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695976.083389:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3877 00000100:00080000:2.0:1787695976.083394:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695976.083402:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695976.083410:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695976.083426:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134164224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695976.083455:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695976.083462:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695976.083467:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695976.083479:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3877 00000100:00080000:2.0:1787695976.083488:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695976.083495:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787695976.083500:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695976.083515:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134164352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787695976.658999:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695981.778998:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695982.803059:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787695984.275201:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787695984.275256:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787695984.275280:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695984.275333:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695984.275352:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695984.275370:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695984.275389:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695984.275408:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787695984.275429:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787695987.923042:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695988.947040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787695994.067011:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787695995.091078:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696000.211074:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696001.234989:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696006.355025:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696006.802966:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696006.802976:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696006.802980:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696006.802990:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3908 00000100:00080000:2.0:1787696006.803000:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696006.803008:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696006.803042:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134168960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696006.803067:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696006.803071:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696006.803073:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696006.803077:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3908 00000100:00080000:2.0:1787696006.803080:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696006.803084:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696006.803094:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696006.803107:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134169088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696006.803121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696006.803135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696006.803138:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696006.803143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3908 00000100:00080000:2.0:1787696006.803147:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696006.803152:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696006.803158:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696006.803169:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134169216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696006.803182:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696006.803186:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696006.803189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696006.803194:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3908 00000100:00080000:2.0:1787696006.803199:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696006.803203:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696006.803207:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696006.803218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134169344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696006.803227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696006.803232:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696006.803235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696006.803245:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3908 00000100:00080000:2.0:1787696006.803252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696006.803257:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696006.803263:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696006.803282:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134169472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696006.803301:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696006.803305:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696006.803307:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696006.803312:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3908 00000100:00080000:2.0:1787696006.803315:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696006.803333:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696006.803337:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696006.803358:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134169600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696006.803373:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696006.803379:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696006.803383:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696006.803388:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3908 00000100:00080000:2.0:1787696006.803392:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696006.803398:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696006.803407:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696006.803430:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134169728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787696007.378975:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696012.499023:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696013.523034:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696018.643055:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696019.667027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696022.162980:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696022.162988:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696022.162992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696022.163001:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3923 00000100:00080000:2.0:1787696022.163011:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696022.163018:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696022.163077:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134171648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696022.163101:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696022.163104:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696022.163105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696022.163109:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3923 00000100:00080000:2.0:1787696022.163112:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696022.163116:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696022.163127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696022.163142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134171776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696022.163164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696022.163168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696022.163170:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696022.163174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3923 00000100:00080000:2.0:1787696022.163178:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696022.163181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696022.163186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696022.163196:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134171904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696022.163208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696022.163210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696022.163211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696022.163214:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3923 00000100:00080000:2.0:1787696022.163220:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696022.163225:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696022.163229:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696022.163239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134172032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696022.163245:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696022.163248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696022.163249:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696022.163252:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3923 00000100:00080000:2.0:1787696022.163255:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696022.163258:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696022.163264:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696022.163276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134172160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696022.163289:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696022.163297:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696022.163301:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696022.163307:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3923 00000100:00080000:2.0:1787696022.163313:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696022.163319:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696022.163327:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696022.163335:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134172288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696022.163343:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696022.163348:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696022.163352:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696022.163356:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3923 00000100:00080000:2.0:1787696022.163359:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696022.163361:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696022.163365:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696022.163376:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134172416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696024.787055:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696025.811031:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696027.282952:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696027.282961:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696027.282987:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696027.282996:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3929 00000100:00080000:2.0:1787696027.283006:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696027.283012:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696027.283046:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134172544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696027.283070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696027.283073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696027.283075:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696027.283078:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3929 00000100:00080000:2.0:1787696027.283081:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696027.283083:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696027.283092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696027.283101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134172672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696027.283110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696027.283113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696027.283114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696027.283120:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3929 00000100:00080000:2.0:1787696027.283122:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696027.283127:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696027.283134:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696027.283142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134172800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696027.283149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696027.283152:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696027.283153:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696027.283155:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3929 00000100:00080000:2.0:1787696027.283158:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696027.283160:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696027.283164:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696027.283171:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134172928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696027.283177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696027.283180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696027.283182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696027.283184:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3929 00000100:00080000:2.0:1787696027.283186:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696027.283188:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696027.283191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696027.283202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134173056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696027.283215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696027.283219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696027.283221:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696027.283226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3929 00000100:00080000:2.0:1787696027.283230:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696027.283235:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696027.283243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696027.283262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134173184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696027.283295:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696027.283303:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696027.283306:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696027.283313:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3929 00000100:00080000:2.0:1787696027.283319:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696027.283323:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696027.283327:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696027.283335:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134173312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696030.931012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696031.955022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696037.522987:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696037.522995:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696037.522998:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696037.523007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3939 00000100:00080000:2.0:1787696037.523016:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696037.523022:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696037.523062:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134174336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696037.523083:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696037.523086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696037.523088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696037.523091:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3939 00000100:00080000:2.0:1787696037.523093:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696037.523095:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696037.523102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696037.523111:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134174464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696037.523120:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696037.523122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696037.523124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696037.523129:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3939 00000100:00080000:2.0:1787696037.523135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696037.523139:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696037.523144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696037.523152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134174592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696037.523164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696037.523167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696037.523168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696037.523171:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3939 00000100:00080000:2.0:1787696037.523181:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696037.523183:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696037.523187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696037.523198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134174720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696037.523208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696037.523211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696037.523213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696037.523215:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3939 00000100:00080000:2.0:1787696037.523219:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696037.523222:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696037.523226:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696037.523234:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134174848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696037.523246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696037.523250:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696037.523253:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696037.523260:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3939 00000100:00080000:2.0:1787696037.523264:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696037.523269:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696037.523274:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696037.523286:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134174976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696037.523336:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696037.523341:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696037.523343:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696037.523348:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3939 00000100:00080000:2.0:1787696037.523357:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696037.523364:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696037.523371:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696037.523386:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134175104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696038.099049:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696042.642979:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696042.642989:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696042.642994:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696042.643005:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3944 00000100:00080000:2.0:1787696042.643016:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696042.643023:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696042.643063:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134175232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696042.643094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696042.643098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696042.643100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696042.643104:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3944 00000100:00080000:2.0:1787696042.643109:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696042.643113:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696042.643125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696042.643140:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134175360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696042.643153:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696042.643157:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696042.643166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696042.643175:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3944 00000100:00080000:2.0:1787696042.643181:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696042.643191:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696042.643198:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696042.643212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134175488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696042.643222:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696042.643226:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696042.643227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696042.643231:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3944 00000100:00080000:2.0:1787696042.643235:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696042.643240:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696042.643245:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696042.643256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134175616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696042.643266:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696042.643270:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696042.643272:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696042.643276:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3944 00000100:00080000:2.0:1787696042.643280:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696042.643293:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696042.643301:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696042.643314:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134175744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696042.643325:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696042.643330:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696042.643332:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696042.643337:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3944 00000100:00080000:2.0:1787696042.643341:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696042.643346:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696042.643351:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696042.643362:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134175872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696042.643372:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696042.643376:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696042.643382:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696042.643392:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3944 00000100:00080000:2.0:1787696042.643401:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696042.643411:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696042.643419:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696042.643439:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134176000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696044.243050:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696047.762940:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696047.762947:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696047.762950:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696047.762957:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3949 00000100:00080000:2.0:1787696047.762965:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696047.762970:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696047.763005:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134176128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696047.763027:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696047.763030:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696047.763031:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696047.763034:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3949 00000100:00080000:2.0:1787696047.763036:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696047.763038:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696047.763045:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696047.763054:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134176256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696047.763060:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696047.763063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696047.763064:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696047.763066:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3949 00000100:00080000:2.0:1787696047.763068:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696047.763072:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696047.763079:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696047.763084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134176384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696047.763091:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696047.763094:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696047.763096:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696047.763100:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3949 00000100:00080000:2.0:1787696047.763104:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696047.763108:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696047.763113:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696047.763125:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134176512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696047.763134:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696047.763145:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696047.763147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696047.763152:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3949 00000100:00080000:2.0:1787696047.763155:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696047.763158:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696047.763162:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696047.763174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134176640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696047.763181:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696047.763184:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696047.763185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696047.763188:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3949 00000100:00080000:2.0:1787696047.763190:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696047.763193:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696047.763196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696047.763203:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134176768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696047.763213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696047.763217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696047.763218:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696047.763222:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3949 00000100:00080000:2.0:1787696047.763225:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696047.763227:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696047.763230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696047.763237:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134176896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696050.387003:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696056.531002:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696062.675018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696068.242935:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696068.242944:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696068.242947:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696068.242955:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3970 00000100:00080000:2.0:1787696068.242964:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696068.242970:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696068.243001:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134179712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696068.243021:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696068.243025:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696068.243028:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696068.243035:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3970 00000100:00080000:2.0:1787696068.243040:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696068.243044:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696068.243055:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696068.243082:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134179840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696068.243092:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696068.243095:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696068.243096:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696068.243099:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3970 00000100:00080000:2.0:1787696068.243112:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696068.243115:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696068.243120:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696068.243131:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134179968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696068.243148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696068.243151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696068.243156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696068.243159:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3970 00000100:00080000:2.0:1787696068.243161:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696068.243164:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696068.243168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696068.243175:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134180096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696068.243182:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696068.243185:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696068.243188:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696068.243198:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3970 00000100:00080000:2.0:1787696068.243203:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696068.243212:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696068.243219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696068.243234:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134180224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696068.243248:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696068.243251:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696068.243253:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696068.243267:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3970 00000100:00080000:2.0:1787696068.243270:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696068.243274:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696068.243277:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696068.243288:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134180352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696068.243297:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696068.243300:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696068.243301:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696068.243304:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3970 00000100:00080000:2.0:1787696068.243308:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696068.243310:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696068.243313:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696068.243324:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134180480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696068.819111:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696073.362990:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696073.362998:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696073.363001:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696073.363009:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3975 00000100:00080000:0.0:1787696073.363017:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696073.363023:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696073.363054:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134180608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696073.363075:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696073.363080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696073.363082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696073.363084:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3975 00000100:00080000:0.0:1787696073.363087:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696073.363089:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696073.363096:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696073.363104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134180736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696073.363111:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696073.363114:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696073.363115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696073.363118:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3975 00000100:00080000:0.0:1787696073.363120:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696073.363122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696073.363125:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696073.363131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134180864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696073.363140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696073.363147:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696073.363150:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696073.363153:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3975 00000100:00080000:0.0:1787696073.363155:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696073.363157:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696073.363160:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696073.363187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134180992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696073.363198:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696073.363200:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696073.363202:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696073.363205:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3975 00000100:00080000:0.0:1787696073.363208:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696073.363211:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696073.363216:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696073.363226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134181120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696073.363239:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696073.363242:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696073.363243:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696073.363246:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3975 00000100:00080000:0.0:1787696073.363255:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696073.363258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696073.363261:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696073.363268:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134181248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696073.363275:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696073.363279:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696073.363282:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696073.363288:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3975 00000100:00080000:0.0:1787696073.363292:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696073.363298:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696073.363305:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696073.363314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134181376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696074.963035:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696078.482953:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696078.482961:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696078.482964:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696078.482973:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3980 00000100:00080000:0.0:1787696078.482982:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696078.482989:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696078.483033:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134181504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696078.483062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696078.483065:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696078.483067:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696078.483070:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3980 00000100:00080000:0.0:1787696078.483073:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696078.483076:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696078.483105:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696078.483117:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134181632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696078.483130:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696078.483135:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696078.483137:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696078.483150:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3980 00000100:00080000:0.0:1787696078.483153:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696078.483156:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696078.483161:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696078.483170:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134181760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696078.483177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696078.483180:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696078.483182:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696078.483184:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3980 00000100:00080000:0.0:1787696078.483187:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696078.483190:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696078.483193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696078.483201:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134181888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696078.483207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696078.483212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696078.483217:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696078.483221:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3980 00000100:00080000:0.0:1787696078.483224:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696078.483227:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696078.483230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696078.483238:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134182016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696078.483244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696078.483246:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696078.483247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696078.483249:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3980 00000100:00080000:0.0:1787696078.483252:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696078.483254:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696078.483260:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696078.483273:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134182144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696078.483298:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696078.483302:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696078.483306:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696078.483316:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3980 00000100:00080000:0.0:1787696078.483323:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696078.483328:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696078.483333:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696078.483347:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134182272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696081.106982:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696087.251034:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696088.722973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696088.722983:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696088.722987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696088.722997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3990 00000100:00080000:0.0:1787696088.723008:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696088.723015:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696088.723057:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134183296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696088.723083:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696088.723086:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696088.723088:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696088.723091:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3990 00000100:00080000:0.0:1787696088.723094:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696088.723097:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696088.723105:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696088.723114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134183424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696088.723123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696088.723126:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696088.723129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696088.723134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3990 00000100:00080000:0.0:1787696088.723141:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696088.723143:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696088.723147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696088.723156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134183552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696088.723163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696088.723166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696088.723167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696088.723170:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3990 00000100:00080000:0.0:1787696088.723184:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696088.723187:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696088.723191:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696088.723202:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134183680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696088.723210:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696088.723213:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696088.723214:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696088.723217:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3990 00000100:00080000:0.0:1787696088.723219:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696088.723222:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696088.723225:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696088.723235:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134183808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696088.723267:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696088.723272:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696088.723277:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696088.723287:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3990 00000100:00080000:0.0:1787696088.723295:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696088.723305:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696088.723313:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696088.723322:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134183936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696088.723330:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696088.723332:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696088.723335:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696088.723343:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3990 00000100:00080000:0.0:1787696088.723347:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696088.723349:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696088.723353:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696088.723365:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134184064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696093.395009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696093.842932:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696093.842941:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696093.842944:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696093.842953:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 3995 00000100:00080000:2.0:1787696093.842962:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696093.842968:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696093.843000:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134184192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696093.843021:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696093.843024:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696093.843025:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696093.843028:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3995 00000100:00080000:2.0:1787696093.843030:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696093.843033:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696093.843040:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696093.843049:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134184320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696093.843057:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696093.843059:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696093.843063:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696093.843067:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3995 00000100:00080000:2.0:1787696093.843072:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696093.843075:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696093.843078:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696093.843087:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134184448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696093.843093:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696093.843096:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696093.843097:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696093.843100:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 3995 00000100:00080000:2.0:1787696093.843102:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696093.843105:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696093.843108:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696093.843115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134184576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696093.843121:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696093.843124:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696093.843127:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696093.843135:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3995 00000100:00080000:2.0:1787696093.843146:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696093.843150:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696093.843153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696093.843165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134184704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696093.843182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696093.843186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696093.843188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696093.843193:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3995 00000100:00080000:2.0:1787696093.843196:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696093.843203:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696093.843210:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696093.843224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134184832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696093.843232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696093.843234:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696093.843236:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696093.843238:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 3995 00000100:00080000:2.0:1787696093.843240:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696093.843243:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696093.843246:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696093.843254:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134184960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696098.962940:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696098.962948:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696098.962952:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696098.962960:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4000 00000100:00080000:2.0:1787696098.962970:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696098.962978:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696098.963019:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134185088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696098.963042:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696098.963045:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696098.963047:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696098.963050:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4000 00000100:00080000:2.0:1787696098.963053:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696098.963057:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696098.963065:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696098.963075:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134185216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696098.963084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696098.963088:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696098.963090:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696098.963098:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4000 00000100:00080000:2.0:1787696098.963103:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696098.963107:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696098.963113:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696098.963125:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134185344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696098.963136:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696098.963140:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696098.963142:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696098.963147:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4000 00000100:00080000:2.0:1787696098.963151:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696098.963154:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696098.963159:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696098.963171:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134185472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696098.963181:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696098.963184:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696098.963185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696098.963189:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4000 00000100:00080000:2.0:1787696098.963192:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696098.963202:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696098.963207:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696098.963221:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134185600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696098.963249:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696098.963254:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696098.963258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696098.963268:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4000 00000100:00080000:2.0:1787696098.963276:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696098.963285:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696098.963293:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696098.963306:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134185728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696098.963313:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696098.963316:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696098.963317:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696098.963321:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4000 00000100:00080000:2.0:1787696098.963326:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696098.963333:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696098.963336:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696098.963345:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134185856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696099.539001:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696104.082994:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696104.083002:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696104.083006:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696104.083014:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4005 00000100:00080000:0.0:1787696104.083023:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696104.083029:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696104.083062:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134185984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696104.083087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696104.083090:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696104.083097:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696104.083100:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4005 00000100:00080000:0.0:1787696104.083103:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696104.083105:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696104.083113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696104.083126:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134186112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696104.083139:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696104.083146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696104.083150:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696104.083155:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4005 00000100:00080000:0.0:1787696104.083161:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696104.083165:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696104.083171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696104.083184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134186240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696104.083192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696104.083194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696104.083196:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696104.083202:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4005 00000100:00080000:0.0:1787696104.083207:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696104.083220:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696104.083225:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696104.083233:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134186368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696104.083241:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696104.083243:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696104.083245:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696104.083250:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4005 00000100:00080000:0.0:1787696104.083255:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696104.083261:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696104.083271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696104.083281:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134186496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696104.083308:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696104.083313:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696104.083317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696104.083322:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4005 00000100:00080000:0.0:1787696104.083328:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696104.083332:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696104.083335:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696104.083342:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134186624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696104.083358:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696104.083360:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696104.083361:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696104.083365:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4005 00000100:00080000:0.0:1787696104.083374:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696104.083379:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696104.083383:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696104.083393:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134186752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696105.683012:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696109.202941:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696109.202950:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696109.202953:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696109.202962:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4010 00000100:00080000:0.0:1787696109.202971:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696109.202977:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696109.203012:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134186880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696109.203043:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696109.203048:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696109.203050:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696109.203054:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4010 00000100:00080000:0.0:1787696109.203059:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696109.203063:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696109.203072:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696109.203084:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134187008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696109.203093:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696109.203095:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696109.203096:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696109.203099:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4010 00000100:00080000:0.0:1787696109.203102:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696109.203107:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696109.203111:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696109.203124:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134187136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696109.203135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696109.203139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696109.203140:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696109.203144:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4010 00000100:00080000:0.0:1787696109.203148:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696109.203152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696109.203157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696109.203169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134187264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696109.203178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696109.203185:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696109.203189:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696109.203194:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4010 00000100:00080000:0.0:1787696109.203198:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696109.203202:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696109.203206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696109.203217:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134187392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696109.203224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696109.203227:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696109.203228:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696109.203234:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4010 00000100:00080000:0.0:1787696109.203239:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696109.203246:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696109.203252:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696109.203262:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134187520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696109.203284:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696109.203290:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696109.203293:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696109.203300:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4010 00000100:00080000:0.0:1787696109.203305:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696109.203309:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696109.203314:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696109.203327:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134187648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696111.827054:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696114.322951:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696114.322959:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696114.322963:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696114.322972:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4016 00000100:00080000:0.0:1787696114.322983:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696114.322991:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696114.323033:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134187776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696114.323055:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696114.323057:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696114.323059:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696114.323062:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4016 00000100:00080000:0.0:1787696114.323065:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696114.323068:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696114.323076:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696114.323086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134187904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696114.323094:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696114.323097:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696114.323101:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696114.323108:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4016 00000100:00080000:0.0:1787696114.323111:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696114.323113:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696114.323117:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696114.323124:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134188032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696114.323133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696114.323137:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696114.323139:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696114.323142:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4016 00000100:00080000:0.0:1787696114.323147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696114.323152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696114.323157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696114.323170:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134188160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696114.323180:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696114.323184:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696114.323185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696114.323188:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4016 00000100:00080000:0.0:1787696114.323192:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696114.323196:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696114.323202:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696114.323217:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134188288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696114.323228:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696114.323234:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696114.323238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696114.323245:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4016 00000100:00080000:0.0:1787696114.323249:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696114.323253:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696114.323258:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696114.323271:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134188416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696114.323282:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696114.323291:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696114.323293:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696114.323297:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4016 00000100:00080000:0.0:1787696114.323301:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696114.323305:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696114.323310:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696114.323325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134188544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696114.899007:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696117.395163:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696117.395209:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696117.395226:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696117.395265:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696117.395276:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696117.395286:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696117.395297:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696117.395306:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696117.395315:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787696117.971040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696119.442991:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696119.443000:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696119.443003:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696119.443010:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4021 00000100:00080000:2.0:1787696119.443020:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696119.443026:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696119.443054:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134188672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696119.443076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696119.443078:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696119.443080:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696119.443082:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4021 00000100:00080000:2.0:1787696119.443085:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696119.443087:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696119.443095:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696119.443108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134188800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696119.443119:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696119.443123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696119.443125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696119.443130:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4021 00000100:00080000:2.0:1787696119.443134:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696119.443138:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696119.443147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696119.443165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134188928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696119.443177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696119.443181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696119.443183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696119.443187:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4021 00000100:00080000:2.0:1787696119.443191:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696119.443195:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696119.443200:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696119.443217:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134189056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696119.443230:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696119.443233:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696119.443235:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696119.443239:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4021 00000100:00080000:2.0:1787696119.443243:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696119.443247:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696119.443254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696119.443269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134189184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696119.443288:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696119.443291:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696119.443294:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696119.443301:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4021 00000100:00080000:2.0:1787696119.443308:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696119.443314:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696119.443320:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696119.443335:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134189312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696119.443356:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696119.443364:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696119.443367:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696119.443371:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4021 00000100:00080000:2.0:1787696119.443409:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696119.443416:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696119.443422:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696119.443440:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134189440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787696120.018992:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696121.043030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696124.115007:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696126.163027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696127.187084:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696130.259095:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696132.307006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696133.331055:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696136.403012:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696138.451070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696139.475022:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696139.922975:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696139.922983:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696139.922987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696139.922995:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4041 00000100:00080000:2.0:1787696139.923003:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696139.923009:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696139.923057:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134192256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696139.923078:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696139.923080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696139.923082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696139.923085:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4041 00000100:00080000:2.0:1787696139.923088:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696139.923093:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696139.923103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696139.923117:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134192384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696139.923129:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696139.923135:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696139.923138:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696139.923146:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4041 00000100:00080000:2.0:1787696139.923151:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696139.923155:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696139.923162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696139.923172:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134192512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696139.923179:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696139.923182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696139.923183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696139.923186:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4041 00000100:00080000:2.0:1787696139.923189:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696139.923192:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696139.923202:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696139.923210:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134192640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696139.923217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696139.923219:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696139.923220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696139.923223:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4041 00000100:00080000:2.0:1787696139.923225:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696139.923231:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696139.923237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696139.923244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134192768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696139.923252:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696139.923255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696139.923257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696139.923263:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4041 00000100:00080000:2.0:1787696139.923266:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696139.923269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696139.923273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696139.923284:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134192896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696139.923307:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696139.923314:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696139.923315:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696139.923318:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4041 00000100:00080000:2.0:1787696139.923321:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696139.923324:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696139.923328:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696139.923336:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134193024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696142.547061:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696144.595078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696145.042994:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696145.043004:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696145.043009:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696145.043019:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4046 00000100:00080000:0.0:1787696145.043029:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696145.043037:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696145.043075:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134193152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696145.043102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696145.043106:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696145.043118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696145.043124:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4046 00000100:00080000:0.0:1787696145.043128:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696145.043133:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696145.043142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696145.043156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134193280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696145.043165:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696145.043168:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696145.043169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696145.043172:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4046 00000100:00080000:0.0:1787696145.043175:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696145.043177:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696145.043181:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696145.043188:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134193408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696145.043202:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696145.043209:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696145.043215:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696145.043221:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4046 00000100:00080000:0.0:1787696145.043226:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696145.043231:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696145.043237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696145.043251:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134193536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696145.043268:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696145.043274:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696145.043277:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696145.043286:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4046 00000100:00080000:0.0:1787696145.043296:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696145.043307:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696145.043314:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696145.043327:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134193664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696145.043338:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696145.043341:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696145.043343:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696145.043347:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4046 00000100:00080000:0.0:1787696145.043352:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696145.043355:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696145.043360:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696145.043371:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134193792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696145.043386:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696145.043392:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696145.043395:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696145.043402:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4046 00000100:00080000:0.0:1787696145.043406:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696145.043411:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696145.043415:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696145.043431:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134193920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696145.619040:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696148.691020:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696150.162937:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696150.162948:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696150.162952:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696150.162963:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4051 00000100:00080000:0.0:1787696150.162980:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696150.162988:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696150.163030:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134194048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696150.163057:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696150.163061:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696150.163063:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696150.163066:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4051 00000100:00080000:0.0:1787696150.163068:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696150.163071:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696150.163082:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696150.163094:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134194176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696150.163108:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696150.163115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696150.163117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696150.163122:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4051 00000100:00080000:0.0:1787696150.163126:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696150.163131:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696150.163137:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696150.163150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134194304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696150.163159:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696150.163163:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696150.163165:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696150.163169:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4051 00000100:00080000:0.0:1787696150.163173:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696150.163178:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696150.163187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696150.163202:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134194432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696150.163211:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696150.163213:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696150.163215:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696150.163218:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4051 00000100:00080000:0.0:1787696150.163221:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696150.163225:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696150.163231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696150.163248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134194560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696150.163279:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696150.163285:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696150.163288:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696150.163295:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4051 00000100:00080000:0.0:1787696150.163300:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696150.163303:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696150.163315:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696150.163329:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134194688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696150.163337:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696150.163343:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696150.163345:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696150.163349:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4051 00000100:00080000:0.0:1787696150.163351:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696150.163354:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696150.163356:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696150.163374:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134194816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696150.739056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696151.763009:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696154.835000:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696155.282939:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696155.282947:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696155.282951:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696155.282961:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4057 00000100:00080000:0.0:1787696155.282970:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696155.282977:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696155.283011:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134194944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696155.283031:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696155.283034:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696155.283036:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696155.283040:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4057 00000100:00080000:0.0:1787696155.283043:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696155.283046:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696155.283054:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696155.283064:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134195072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696155.283086:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696155.283090:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696155.283091:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696155.283095:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4057 00000100:00080000:0.0:1787696155.283098:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696155.283101:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696155.283106:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696155.283115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134195200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696155.283123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696155.283127:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696155.283129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696155.283135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4057 00000100:00080000:0.0:1787696155.283155:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696155.283158:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696155.283163:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696155.283174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134195328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696155.283191:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696155.283205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696155.283207:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696155.283215:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4057 00000100:00080000:0.0:1787696155.283221:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696155.283368:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696155.283376:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696155.283396:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134195456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696155.283414:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696155.283418:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696155.283420:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696155.283424:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4057 00000100:00080000:0.0:1787696155.283429:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696155.283433:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696155.283439:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696155.283454:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134195584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696155.283467:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696155.283473:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696155.283475:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696155.283480:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4057 00000100:00080000:0.0:1787696155.283483:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696155.283488:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696155.283494:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696155.283504:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134195712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696156.882990:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696157.906993:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696160.402947:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696160.402957:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696160.402967:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696160.402976:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4062 00000100:00080000:0.0:1787696160.402986:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696160.402992:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696160.403027:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134195840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696160.403057:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696160.403060:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696160.403062:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696160.403066:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4062 00000100:00080000:0.0:1787696160.403069:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696160.403073:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696160.403082:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696160.403100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134195968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696160.403119:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696160.403122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696160.403123:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696160.403126:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4062 00000100:00080000:0.0:1787696160.403129:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696160.403132:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696160.403137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696160.403146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134196096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696160.403154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696160.403157:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696160.403158:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696160.403161:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4062 00000100:00080000:0.0:1787696160.403163:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696160.403166:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696160.403170:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696160.403189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134196224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696160.403197:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696160.403200:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696160.403202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696160.403205:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4062 00000100:00080000:0.0:1787696160.403208:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696160.403211:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696160.403215:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696160.403223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134196352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696160.403230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696160.403232:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696160.403234:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696160.403238:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4062 00000100:00080000:0.0:1787696160.403240:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696160.403244:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696160.403248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696160.403255:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134196480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696160.403264:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696160.403266:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696160.403268:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696160.403271:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4062 00000100:00080000:0.0:1787696160.403274:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696160.403277:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696160.403281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696160.403289:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134196608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696160.978991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696163.027023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696164.051089:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696165.522941:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696165.522955:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696165.522960:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696165.522973:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4067 00000100:00080000:0.0:1787696165.522987:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696165.522996:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696165.523041:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134196736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696165.523069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696165.523072:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696165.523073:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696165.523076:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4067 00000100:00080000:0.0:1787696165.523079:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696165.523081:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696165.523092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696165.523105:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134196864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696165.523123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696165.523128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696165.523130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696165.523134:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4067 00000100:00080000:0.0:1787696165.523137:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696165.523140:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696165.523145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696165.523154:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134196992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696165.523161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696165.523163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696165.523165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696165.523167:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4067 00000100:00080000:0.0:1787696165.523171:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696165.523174:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696165.523179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696165.523191:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134197120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696165.523207:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696165.523211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696165.523213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696165.523218:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4067 00000100:00080000:0.0:1787696165.523229:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696165.523233:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696165.523238:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696165.523250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134197248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696165.523272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696165.523277:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696165.523279:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696165.523287:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4067 00000100:00080000:0.0:1787696165.523294:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696165.523299:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696165.523304:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696165.523316:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134197376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696165.523335:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696165.523338:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696165.523339:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696165.523343:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4067 00000100:00080000:0.0:1787696165.523347:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696165.523352:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696165.523358:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696165.523373:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134197504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696167.123007:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696169.170974:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696170.195059:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696173.267000:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696175.315031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696176.339035:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696179.411002:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696181.459072:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696182.482977:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696183.955334:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696183.955378:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696183.955394:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696183.955432:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696183.955441:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696183.955448:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696183.955460:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696183.955470:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696183.955481:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787696184.531123:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696184.531673:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000091b8dc44/0x678ae32e95324000 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d913b expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.531696:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.531699:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.531713:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c2efdf95/0x678ae32e95323fe4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d90ee expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.531742:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x23:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.531758:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x23:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.531777:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002d024e86/0x678ae32e95323fd6 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d90cb expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.531791:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.531811:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.531829:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c8ab5e32/0x678ae32e95323fba lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d907e expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.531842:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x5d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.531899:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x5d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.531928:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000005eea239d/0x678ae32e95323f9e lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d9038 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.531979:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.531985:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.531994:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000035f15bfc/0x678ae32e95323f66 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8f97 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532003:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x63:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532016:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x63:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532035:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002839e049/0x678ae32e95323f58 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8f74 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532057:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x28:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532064:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x28:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532075:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000006a3c2af6/0x678ae32e95323f43 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8f35 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532114:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532120:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532127:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000000018e022/0x678ae32e95323f35 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8f12 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532135:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532140:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532150:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c81d5850/0x678ae32e95323f19 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x23:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8ec5 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532191:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x5b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532201:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x5b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532216:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000264cf771/0x678ae32e95323ef6 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8e63 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532278:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x65:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532297:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x65:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532312:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000003feec003/0x678ae32e95323ebe lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x814:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8dc9 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532330:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xa3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532336:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xa3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532371:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000ff1c8366/0x678ae32e95323e78 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8d05 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532419:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532428:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532444:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000008f66c76e/0x678ae32e95323e6a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8ce2 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532460:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532468:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532483:0:14499:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000009126e814/0x678ae32e95323e5c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x297ceef0832d8cbf expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787696184.532520:0:14499:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x8f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787696184.532543:0:14499:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x8f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787696184.532593:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000003e94daf0/0x678ae32e953241f8 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d9651 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532600:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002b67edb8/0x678ae32e953241b2 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x1c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d958d expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532605:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000ce2fe0a2/0x678ae32e95324150 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x813:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d948a expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532609:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002c36f918/0x678ae32e95324142 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d9467 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532613:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000daa1b329/0x678ae32e95324111 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d93fe expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532617:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000069ac0239/0x678ae32e95324103 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d93d4 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532621:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000201f969c/0x678ae32e95324093 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d92ae expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532627:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000007599c529/0x678ae32e9532405b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d921b expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532657:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000009a6292f3/0x678ae32e9532404d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d91f8 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532681:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000006de19dc7/0x678ae32e95324023 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d918f expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532730:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000091b8dc44/0x678ae32e95324000 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d913b expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532755:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c2efdf95/0x678ae32e95323fe4 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d90ee expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532780:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002d024e86/0x678ae32e95323fd6 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d90cb expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532803:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c8ab5e32/0x678ae32e95323fba lrc: 1/0,0 mode: --/PR res: [0x280000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d907e expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532839:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000005eea239d/0x678ae32e95323f9e lrc: 1/0,0 mode: --/PR res: [0x280000400:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d9038 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532900:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000035f15bfc/0x678ae32e95323f66 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f97 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532923:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002839e049/0x678ae32e95323f58 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f74 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532935:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000006a3c2af6/0x678ae32e95323f43 lrc: 1/0,0 mode: --/PR res: [0x280000400:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f35 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532972:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000000018e022/0x678ae32e95323f35 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f12 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.532996:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c81d5850/0x678ae32e95323f19 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x23:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8ec5 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533016:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000264cf771/0x678ae32e95323ef6 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8e63 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533035:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000003feec003/0x678ae32e95323ebe lrc: 1/0,0 mode: --/PR res: [0x280000400:0x814:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8dc9 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533076:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000ff1c8366/0x678ae32e95323e78 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8d05 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533112:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000008f66c76e/0x678ae32e95323e6a lrc: 1/0,0 mode: --/PR res: [0x280000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8ce2 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533165:0:14499:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000009126e814/0x678ae32e95323e5c lrc: 1/0,0 mode: --/PR res: [0x280000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8cbf expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533208:0:14499:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 25 locks packed 00010000:00010000:3.0:1787696184.533238:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000003e94daf0/0x678ae32e953241f8 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d9651 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00080000:2.0:1787696184.533288:0:2362:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9d85c0fb1180 x1874531134200320/t0(0) o103->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:17/18 lens 512/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1787696184.533289:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002b67edb8/0x678ae32e953241b2 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x1c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d958d expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00080000:0.0:1787696184.533351:0:2360:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9d85b9d2ea00 x1874531134200448/t0(0) o103->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:17/18 lens 496/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1787696184.533355:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000ce2fe0a2/0x678ae32e95324150 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x813:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d948a expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533395:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002c36f918/0x678ae32e95324142 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d9467 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533452:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000daa1b329/0x678ae32e95324111 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d93fe expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533510:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000069ac0239/0x678ae32e95324103 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d93d4 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533597:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000201f969c/0x678ae32e95324093 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d92ae expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533675:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000007599c529/0x678ae32e9532405b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d921b expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533735:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000009a6292f3/0x678ae32e9532404d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d91f8 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533787:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000006de19dc7/0x678ae32e95324023 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d918f expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533837:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000091b8dc44/0x678ae32e95324000 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d913b expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533891:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c2efdf95/0x678ae32e95323fe4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d90ee expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533915:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002d024e86/0x678ae32e95323fd6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d90cb expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.533954:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c8ab5e32/0x678ae32e95323fba lrc: 0/0,0 mode: --/PR res: [0x280000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d907e expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534003:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000005eea239d/0x678ae32e95323f9e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d9038 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534061:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 0000000035f15bfc/0x678ae32e95323f66 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f97 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534099:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000002839e049/0x678ae32e95323f58 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f74 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534154:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000006a3c2af6/0x678ae32e95323f43 lrc: 0/0,0 mode: --/PR res: [0x280000400:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f35 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534212:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000000018e022/0x678ae32e95323f35 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8f12 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534231:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000c81d5850/0x678ae32e95323f19 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x23:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8ec5 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534264:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000264cf771/0x678ae32e95323ef6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8e63 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534283:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000003feec003/0x678ae32e95323ebe lrc: 0/0,0 mode: --/PR res: [0x280000400:0x814:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8dc9 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534307:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 00000000ff1c8366/0x678ae32e95323e78 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8d05 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534325:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000008f66c76e/0x678ae32e95323e6a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8ce2 expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534344:0:14499:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d858c4eb000 lock: 000000009126e814/0x678ae32e95323e5c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x297ceef0832d8cbf expref: -99 pid: 96400 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787696184.534373:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696184.534377:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696184.534380:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696185.555038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696186.002989:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696186.003000:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696186.003004:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696186.003016:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4087 00000100:00080000:3.0:1787696186.003030:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696186.003040:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696186.003083:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134200576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696186.003112:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696186.003115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696186.003117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696186.003120:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4087 00000100:00080000:3.0:1787696186.003122:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696186.003124:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696186.003135:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696186.003148:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134200704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696186.003159:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696186.003162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696186.003164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696186.003169:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4087 00000100:00080000:3.0:1787696186.003173:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696186.003177:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696186.003183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696186.003198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134200832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696186.003213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696186.003216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696186.003219:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696186.003223:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4087 00000100:00080000:3.0:1787696186.003260:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696186.003266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696186.003272:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696186.003287:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134200960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696186.003297:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696186.003301:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696186.003302:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696186.003311:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4087 00000100:00080000:3.0:1787696186.003316:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696186.003325:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696186.003332:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696186.003347:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134201088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696186.003357:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696186.003361:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696186.003367:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696186.003372:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4087 00000100:00080000:3.0:1787696186.003376:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696186.003381:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696186.003387:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696186.003401:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134201216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696186.003416:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696186.003426:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696186.003431:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696186.003439:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4087 00000100:00080000:3.0:1787696186.003446:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696186.003456:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696186.003465:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696186.003486:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134201344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696186.579061:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696187.603051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696188.626976:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696190.675044:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696191.122955:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696191.122963:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696191.122966:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696191.122975:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4092 00000100:00080000:3.0:1787696191.122983:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696191.122989:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696191.123028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134201472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696191.123050:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696191.123054:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696191.123056:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696191.123061:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4092 00000100:00080000:3.0:1787696191.123065:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696191.123070:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696191.123081:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696191.123098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134201600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696191.123112:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696191.123116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696191.123118:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696191.123123:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4092 00000100:00080000:3.0:1787696191.123127:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696191.123132:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696191.123153:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696191.123168:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134201728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696191.123186:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696191.123190:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696191.123193:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696191.123199:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4092 00000100:00080000:3.0:1787696191.123205:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696191.123210:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696191.123216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696191.123233:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134201856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696191.123247:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696191.123250:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696191.123253:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696191.123261:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4092 00000100:00080000:3.0:1787696191.123269:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696191.123276:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696191.123284:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696191.123303:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134201984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696191.123319:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696191.123325:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696191.123327:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696191.123333:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4092 00000100:00080000:3.0:1787696191.123339:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696191.123349:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696191.123356:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696191.123374:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134202112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696191.123413:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696191.123416:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696191.123419:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696191.123424:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4092 00000100:00080000:3.0:1787696191.123429:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696191.123432:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696191.123438:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696191.123455:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134202240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696191.699037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696192.723081:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696193.747028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696194.771028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696196.242927:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696196.242938:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696196.242942:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696196.242952:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4098 00000100:00080000:3.0:1787696196.242962:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696196.242968:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696196.243000:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134202368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696196.243025:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696196.243028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696196.243030:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696196.243033:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4098 00000100:00080000:3.0:1787696196.243035:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696196.243038:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696196.243045:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696196.243067:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134202496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696196.243078:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696196.243082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696196.243084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696196.243089:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4098 00000100:00080000:3.0:1787696196.243091:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696196.243097:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696196.243102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696196.243112:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134202624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696196.243120:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696196.243123:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696196.243124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696196.243126:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4098 00000100:00080000:3.0:1787696196.243128:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696196.243130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696196.243133:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696196.243143:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134202752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696196.243149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696196.243152:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696196.243153:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696196.243156:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4098 00000100:00080000:3.0:1787696196.243158:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696196.243161:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696196.243168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696196.243177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134202880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696196.243184:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696196.243186:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696196.243188:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696196.243190:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4098 00000100:00080000:3.0:1787696196.243193:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696196.243196:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696196.243199:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696196.243205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134203008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696196.243211:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696196.243216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696196.243220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696196.243227:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4098 00000100:00080000:3.0:1787696196.243233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696196.243236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696196.243241:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696196.243252:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134203136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696196.819034:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696197.843006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696198.867033:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696199.891035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696200.915087:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696201.362913:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696201.362928:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696201.362934:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696201.362942:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4103 00000100:00080000:3.0:1787696201.362951:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696201.362957:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696201.362999:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134203264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696201.363019:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696201.363022:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696201.363024:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696201.363026:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4103 00000100:00080000:3.0:1787696201.363029:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696201.363031:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696201.363039:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696201.363047:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134203392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696201.363054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696201.363057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696201.363059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696201.363061:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4103 00000100:00080000:3.0:1787696201.363063:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696201.363067:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696201.363072:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696201.363086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134203520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696201.363097:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696201.363101:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696201.363104:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696201.363107:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4103 00000100:00080000:3.0:1787696201.363111:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696201.363115:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696201.363121:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696201.363133:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134203648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696201.363144:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696201.363149:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696201.363152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696201.363160:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4103 00000100:00080000:3.0:1787696201.363165:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696201.363170:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696201.363175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696201.363187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134203776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696201.363198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696201.363202:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696201.363206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696201.363215:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4103 00000100:00080000:3.0:1787696201.363223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696201.363236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696201.363246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696201.363267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134203904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696201.363286:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696201.363291:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696201.363295:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696201.363303:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4103 00000100:00080000:3.0:1787696201.363309:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696201.363313:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696201.363319:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696201.363334:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134204032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696202.963033:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696203.987021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696205.011022:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696206.035019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696206.483093:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134204160:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696206.483163:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696206.483209:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696206/real 1787696206] req@ffff9d85b9d2d500 x1874531134204160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696206.483235:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134204160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696206.483246:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696206.483252:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696206.483259:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134204160:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696206.483284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134204288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696206.483304:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696206.483322:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696206/real 1787696206] req@ffff9d85b9d2f800 x1874531134204288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696206.483354:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134204288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696206.483364:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696206.483368:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696206.483371:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134204288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696206.483376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134204416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696206.483392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696206.483399:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696206/real 1787696206] req@ffff9d85b9d2dc00 x1874531134204416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696206.483412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134204416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696206.483417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696206.483419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696206.483421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134204416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696206.483424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134204544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696206.483434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696206.483439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696206/real 1787696206] req@ffff9d85b9d2c700 x1874531134204544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696206.483446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134204544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696206.483451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696206.483453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696206.483455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134204544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696206.483458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134204672:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696206.483467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696206.483473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696206/real 1787696206] req@ffff9d85b9d2ed80 x1874531134204672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696206.483480:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134204672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696206.483489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696206.483491:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696206.483493:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134204672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696206.483496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134204800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696206.483505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696206.483511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696206/real 1787696206] req@ffff9d85b9d2c380 x1874531134204800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696206.483517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134204800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696206.483522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696206.483523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696206.483525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134204800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696206.483537:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134204928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696206.483547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696206.483553:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696206/real 1787696206] req@ffff9d85b9d2df80 x1874531134204928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696206.483559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2df80 x1874531134204928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696206.483565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696206.483566:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696206.483569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134204928:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696207.059057:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696209.107032:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696210.130979:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696211.154970:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696211.602957:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696211.602967:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696211.602972:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696211.602982:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4113 00000100:00080000:3.0:1787696211.602992:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696211.602999:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696211.603036:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134205056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696211.603065:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696211.603069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696211.603071:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696211.603076:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4113 00000100:00080000:3.0:1787696211.603080:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696211.603085:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696211.603095:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696211.603109:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134205184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696211.603122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696211.603126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696211.603128:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696211.603133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4113 00000100:00080000:3.0:1787696211.603137:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696211.603141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696211.603147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696211.603168:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134205312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696211.603180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696211.603184:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696211.603186:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696211.603190:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4113 00000100:00080000:3.0:1787696211.603194:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696211.603201:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696211.603209:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696211.603226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134205440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696211.603257:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696211.603265:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696211.603268:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696211.603277:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4113 00000100:00080000:3.0:1787696211.603281:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696211.603285:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696211.603289:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696211.603299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134205568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696211.603312:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696211.603317:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696211.603319:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696211.603325:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4113 00000100:00080000:3.0:1787696211.603329:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696211.603336:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696211.603344:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696211.603359:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134205696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696211.603375:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696211.603381:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696211.603385:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696211.603394:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4113 00000100:00080000:3.0:1787696211.603400:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696211.603406:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696211.603413:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696211.603429:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134205824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696212.179009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696213.202991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696215.250999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696216.275051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696216.722977:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696216.722986:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696216.722989:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696216.722997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4118 00000100:00080000:0.0:1787696216.723007:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696216.723013:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696216.723047:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134205952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696216.723069:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696216.723073:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696216.723074:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696216.723077:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4118 00000100:00080000:0.0:1787696216.723080:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696216.723082:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696216.723091:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696216.723101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134206080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696216.723109:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696216.723113:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696216.723117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696216.723123:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4118 00000100:00080000:0.0:1787696216.723125:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696216.723128:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696216.723132:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696216.723141:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134206208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696216.723148:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696216.723150:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696216.723152:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696216.723154:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4118 00000100:00080000:0.0:1787696216.723166:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696216.723168:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696216.723171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696216.723187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134206336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696216.723199:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696216.723201:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696216.723203:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696216.723206:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4118 00000100:00080000:0.0:1787696216.723208:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696216.723213:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696216.723217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696216.723230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134206464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696216.723240:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696216.723247:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696216.723250:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696216.723256:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4118 00000100:00080000:0.0:1787696216.723262:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696216.723268:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696216.723274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696216.723290:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134206592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696216.723310:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696216.723315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696216.723317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696216.723326:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4118 00000100:00080000:0.0:1787696216.723333:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696216.723342:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696216.723350:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696216.723374:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134206720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696217.299047:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696218.323108:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696219.347049:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696221.394970:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696221.843119:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134206848:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696221.843212:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696221.843256:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696221/real 1787696221] req@ffff9d85b9d2e680 x1874531134206848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696221.843273:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134206848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696221.843284:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696221.843288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696221.843292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134206848:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696221.843314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134206976:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696221.843337:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696221.843350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696221/real 1787696221] req@ffff9d85b9d2ca80 x1874531134206976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696221.843363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134206976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696221.843369:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696221.843371:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696221.843373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134206976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696221.843378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134207104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696221.843394:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696221.843401:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696221/real 1787696221] req@ffff9d85b9d2fb80 x1874531134207104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696221.843408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134207104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696221.843418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696221.843420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696221.843422:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134207104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696221.843425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134207232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696221.843435:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696221.843440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696221/real 1787696221] req@ffff9d85b9d2e300 x1874531134207232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696221.843447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134207232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696221.843452:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696221.843454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696221.843456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134207232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696221.843459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134207360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696221.843469:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696221.843475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696221/real 1787696221] req@ffff9d85b9d2ce00 x1874531134207360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696221.843482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134207360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696221.843486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696221.843488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696221.843491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134207360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696221.843494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134207488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696221.843503:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696221.843508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696221/real 1787696221] req@ffff9d85b9d2c000 x1874531134207488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696221.843515:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134207488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696221.843520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696221.843522:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696221.843524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134207488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696221.843536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134207616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696221.843546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696221.843551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696221/real 1787696221] req@ffff9d85b9d2f480 x1874531134207616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696221.843558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134207616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696221.843563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696221.843565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696221.843567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134207616:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696222.419043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696223.443000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696224.466990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696225.490991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696226.962963:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696226.962975:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696226.962979:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696226.962990:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4128 00000100:00080000:3.0:1787696226.963003:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696226.963009:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696226.963042:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134207744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696226.963067:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696226.963069:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696226.963071:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696226.963074:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4128 00000100:00080000:3.0:1787696226.963077:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696226.963080:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696226.963089:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696226.963098:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134207872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696226.963108:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696226.963116:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696226.963118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696226.963121:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4128 00000100:00080000:3.0:1787696226.963123:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696226.963125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696226.963129:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696226.963138:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134208000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696226.963145:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696226.963148:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696226.963149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696226.963151:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4128 00000100:00080000:3.0:1787696226.963154:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696226.963157:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696226.963159:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696226.963166:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134208128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696226.963172:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696226.963178:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696226.963183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696226.963188:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4128 00000100:00080000:3.0:1787696226.963192:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696226.963197:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696226.963202:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696226.963216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134208256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696226.963226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696226.963230:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696226.963232:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696226.963237:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4128 00000100:00080000:3.0:1787696226.963244:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696226.963252:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696226.963260:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696226.963274:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134208384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696226.963287:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696226.963292:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696226.963294:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696226.963302:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4128 00000100:00080000:3.0:1787696226.963305:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696226.963308:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696226.963310:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696226.963316:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134208512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696227.538985:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696228.562988:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696229.586991:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696230.610999:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696231.634995:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696232.082948:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696232.082957:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696232.082961:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696232.082970:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4133 00000100:00080000:3.0:1787696232.082979:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696232.082986:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696232.083027:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134208640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696232.083053:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696232.083057:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696232.083058:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696232.083061:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4133 00000100:00080000:3.0:1787696232.083063:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696232.083066:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696232.083073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696232.083083:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134208768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696232.083090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696232.083093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696232.083094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696232.083097:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4133 00000100:00080000:3.0:1787696232.083098:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696232.083101:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696232.083106:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696232.083114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134208896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696232.083122:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696232.083124:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696232.083125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696232.083128:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4133 00000100:00080000:3.0:1787696232.083131:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696232.083133:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696232.083136:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696232.083145:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134209024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696232.083151:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696232.083154:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696232.083154:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696232.083156:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4133 00000100:00080000:3.0:1787696232.083159:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696232.083161:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696232.083164:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696232.083171:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134209152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696232.083182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696232.083185:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696232.083187:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696232.083191:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4133 00000100:00080000:3.0:1787696232.083194:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696232.083196:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696232.083199:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696232.083206:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134209280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696232.083216:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696232.083218:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696232.083219:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696232.083221:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4133 00000100:00080000:3.0:1787696232.083223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696232.083226:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696232.083229:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696232.083246:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134209408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696233.682989:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696234.707058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696235.731021:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696236.754996:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696237.202955:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696237.202967:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696237.202971:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696237.202982:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4138 00000100:00080000:0.0:1787696237.202996:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696237.203003:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696237.203049:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134209536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696237.203075:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696237.203079:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696237.203080:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696237.203083:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4138 00000100:00080000:0.0:1787696237.203086:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696237.203089:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696237.203098:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696237.203108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134209664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696237.203118:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696237.203122:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696237.203124:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696237.203133:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4138 00000100:00080000:0.0:1787696237.203138:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696237.203142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696237.203145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696237.203164:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134209792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696237.203177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696237.203181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696237.203183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696237.203189:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4138 00000100:00080000:0.0:1787696237.203210:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696237.203215:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696237.203222:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696237.203246:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134209920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696237.203258:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696237.203262:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696237.203265:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696237.203272:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4138 00000100:00080000:0.0:1787696237.203281:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696237.203289:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696237.203304:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696237.203322:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134210048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696237.203340:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696237.203363:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696237.203372:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696237.203381:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4138 00000100:00080000:0.0:1787696237.203386:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696237.203392:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696237.203399:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696237.203424:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134210176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696237.203436:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696237.203447:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696237.203451:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696237.203461:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4138 00000100:00080000:0.0:1787696237.203472:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696237.203482:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696237.203502:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696237.203529:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134210304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696237.779102:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696239.827040:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696240.850991:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696241.875056:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696242.322946:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696242.322955:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696242.322959:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696242.322969:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4144 00000100:00080000:3.0:1787696242.322980:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696242.322987:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696242.323017:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134210432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696242.323036:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696242.323039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696242.323040:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696242.323044:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4144 00000100:00080000:3.0:1787696242.323046:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696242.323049:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696242.323055:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696242.323062:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134210560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696242.323069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696242.323071:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696242.323072:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696242.323075:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4144 00000100:00080000:3.0:1787696242.323078:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696242.323082:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696242.323089:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696242.323098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134210688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696242.323106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696242.323109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696242.323110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696242.323114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4144 00000100:00080000:3.0:1787696242.323117:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696242.323121:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696242.323126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696242.323137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134210816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696242.323149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696242.323151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696242.323152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696242.323156:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4144 00000100:00080000:3.0:1787696242.323159:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696242.323162:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696242.323165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696242.323172:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134210944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696242.323177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696242.323179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696242.323181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696242.323183:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4144 00000100:00080000:3.0:1787696242.323185:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696242.323190:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696242.323203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696242.323212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134211072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696242.323233:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696242.323237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696242.323239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696242.323242:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4144 00000100:00080000:3.0:1787696242.323246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696242.323252:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696242.323268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696242.323280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134211200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696242.899064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696243.923068:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696245.970995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696246.995027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696247.442950:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696247.442959:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696247.442963:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696247.442978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4149 00000100:00080000:3.0:1787696247.442989:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696247.442997:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696247.443028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134211328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696247.443056:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696247.443059:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696247.443061:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696247.443065:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4149 00000100:00080000:3.0:1787696247.443069:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696247.443072:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696247.443082:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696247.443095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134211456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696247.443112:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696247.443115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696247.443117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696247.443122:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4149 00000100:00080000:3.0:1787696247.443126:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696247.443130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696247.443135:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696247.443152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134211584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696247.443206:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696247.443209:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696247.443211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696247.443217:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4149 00000100:00080000:3.0:1787696247.443223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696247.443228:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696247.443234:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696247.443249:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134211712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696247.443259:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696247.443262:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696247.443264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696247.443268:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4149 00000100:00080000:3.0:1787696247.443272:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696247.443276:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696247.443281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696247.443300:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134211840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696247.443314:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696247.443318:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696247.443321:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696247.443327:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4149 00000100:00080000:3.0:1787696247.443331:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696247.443337:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696247.443346:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696247.443366:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134211968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696247.443393:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696247.443396:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696247.443398:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696247.443403:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4149 00000100:00080000:3.0:1787696247.443407:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696247.443411:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696247.443416:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696247.443428:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134212096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696248.019067:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696249.042997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696250.067050:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787696250.515143:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787696250.515192:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787696250.515209:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696250.515257:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696250.515269:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696250.515277:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696250.515287:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696250.515296:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696250.515305:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:3.0:1787696252.114992:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696252.562926:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696252.562934:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696252.562938:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696252.562946:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4154 00000100:00080000:3.0:1787696252.562956:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696252.562962:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696252.562991:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134212224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696252.563011:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696252.563014:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696252.563015:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696252.563017:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4154 00000100:00080000:3.0:1787696252.563020:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696252.563021:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696252.563028:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696252.563036:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134212352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696252.563042:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696252.563044:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696252.563045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696252.563047:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4154 00000100:00080000:3.0:1787696252.563050:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696252.563052:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696252.563056:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696252.563064:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134212480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696252.563074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696252.563087:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696252.563090:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696252.563094:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4154 00000100:00080000:3.0:1787696252.563097:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696252.563100:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696252.563104:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696252.563117:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134212608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696252.563124:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696252.563128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696252.563129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696252.563132:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4154 00000100:00080000:3.0:1787696252.563136:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696252.563142:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696252.563150:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696252.563161:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134212736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696252.563176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696252.563179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696252.563181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696252.563185:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4154 00000100:00080000:3.0:1787696252.563188:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696252.563193:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696252.563198:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696252.563211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134212864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696252.563221:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696252.563227:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696252.563231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696252.563239:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4154 00000100:00080000:3.0:1787696252.563245:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696252.563253:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696252.563261:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696252.563278:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134212992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696253.139009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696254.162972:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696255.187001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696256.210955:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696257.682940:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696257.682948:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696257.682951:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696257.682959:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4159 00000100:00080000:3.0:1787696257.682968:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696257.682973:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696257.683021:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134213120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696257.683041:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696257.683044:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696257.683045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696257.683048:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4159 00000100:00080000:3.0:1787696257.683050:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696257.683052:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696257.683059:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696257.683067:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134213248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696257.683073:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696257.683076:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696257.683077:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696257.683079:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4159 00000100:00080000:3.0:1787696257.683081:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696257.683082:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696257.683086:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696257.683093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134213376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696257.683108:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696257.683110:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696257.683111:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696257.683113:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4159 00000100:00080000:3.0:1787696257.683115:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696257.683117:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696257.683120:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696257.683128:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134213504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696257.683134:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696257.683136:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696257.683137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696257.683139:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4159 00000100:00080000:3.0:1787696257.683141:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696257.683144:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696257.683146:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696257.683154:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134213632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696257.683161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696257.683163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696257.683165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696257.683168:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4159 00000100:00080000:3.0:1787696257.683170:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696257.683173:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696257.683175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696257.683182:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134213760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696257.683186:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696257.683188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696257.683189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696257.683191:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4159 00000100:00080000:3.0:1787696257.683193:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696257.683196:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696257.683199:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696257.683207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134213888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696258.259000:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696259.283037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696260.306992:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696261.331074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696262.355010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696262.802948:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696262.802957:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696262.802960:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696262.802975:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4164 00000100:00080000:3.0:1787696262.802984:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696262.802991:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696262.803023:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134214016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696262.803053:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696262.803057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696262.803059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696262.803064:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4164 00000100:00080000:3.0:1787696262.803068:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696262.803073:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696262.803083:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696262.803097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134214144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696262.803111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696262.803114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696262.803115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696262.803119:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4164 00000100:00080000:3.0:1787696262.803122:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696262.803124:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696262.803128:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696262.803137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134214272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696262.803143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696262.803146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696262.803147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696262.803150:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4164 00000100:00080000:3.0:1787696262.803153:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696262.803156:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696262.803161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696262.803171:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134214400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696262.803179:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696262.803181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696262.803182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696262.803185:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4164 00000100:00080000:3.0:1787696262.803187:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696262.803190:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696262.803200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696262.803210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134214528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696262.803228:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696262.803231:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696262.803232:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696262.803237:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4164 00000100:00080000:3.0:1787696262.803240:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696262.803247:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696262.803255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696262.803264:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134214656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696262.803279:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696262.803282:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696262.803285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696262.803289:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4164 00000100:00080000:3.0:1787696262.803291:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696262.803297:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696262.803302:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696262.803309:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134214784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696264.403092:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696265.427045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696266.451056:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696267.475007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696267.922926:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696267.922936:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696267.922940:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696267.922950:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4169 00000100:00080000:0.0:1787696267.922961:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696267.922968:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696267.923003:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134214912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696267.923028:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696267.923031:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696267.923033:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696267.923037:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4169 00000100:00080000:0.0:1787696267.923041:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696267.923046:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696267.923055:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696267.923070:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134215040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696267.923082:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696267.923086:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696267.923088:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696267.923092:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4169 00000100:00080000:0.0:1787696267.923096:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696267.923099:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696267.923104:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696267.923115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134215168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696267.923125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696267.923129:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696267.923130:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696267.923133:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4169 00000100:00080000:0.0:1787696267.923137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696267.923142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696267.923149:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696267.923164:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134215296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696267.923173:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696267.923176:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696267.923179:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696267.923183:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4169 00000100:00080000:0.0:1787696267.923187:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696267.923195:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696267.923207:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696267.923223:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134215424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696267.923244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696267.923249:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696267.923251:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696267.923256:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4169 00000100:00080000:0.0:1787696267.923260:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696267.923281:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696267.923291:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696267.923308:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134215552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696267.923344:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696267.923354:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696267.923358:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696267.923367:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4169 00000100:00080000:0.0:1787696267.923372:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696267.923380:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696267.923385:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696267.923398:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134215680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696268.498997:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696270.546970:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696271.571040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696272.595020:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696273.042988:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696273.043000:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696273.043004:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696273.043015:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4174 00000100:00080000:3.0:1787696273.043028:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696273.043037:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696273.043071:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134215808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696273.043094:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696273.043098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696273.043101:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696273.043105:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4174 00000100:00080000:3.0:1787696273.043109:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696273.043113:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696273.043123:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696273.043137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134215936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696273.043148:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696273.043151:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696273.043152:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696273.043156:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4174 00000100:00080000:3.0:1787696273.043159:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696273.043162:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696273.043167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696273.043175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134216064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696273.043184:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696273.043187:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696273.043188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696273.043192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4174 00000100:00080000:3.0:1787696273.043194:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696273.043197:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696273.043201:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696273.043211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134216192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696273.043219:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696273.043222:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696273.043224:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696273.043227:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4174 00000100:00080000:3.0:1787696273.043230:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696273.043233:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696273.043237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696273.043246:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134216320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696273.043254:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696273.043258:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696273.043259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696273.043263:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4174 00000100:00080000:3.0:1787696273.043266:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696273.043269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696273.043274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696273.043282:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134216448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696273.043291:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696273.043294:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696273.043296:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696273.043299:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4174 00000100:00080000:3.0:1787696273.043302:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696273.043305:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696273.043310:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696273.043317:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134216576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696273.619004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696274.643089:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696276.691044:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696277.715008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696278.162915:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696278.162924:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696278.162929:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696278.162939:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4179 00000100:00080000:3.0:1787696278.162950:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696278.162957:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696278.162994:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134216704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696278.163024:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696278.163028:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696278.163030:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696278.163034:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4179 00000100:00080000:3.0:1787696278.163037:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696278.163041:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696278.163050:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696278.163063:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134216832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696278.163074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696278.163080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696278.163085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696278.163092:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4179 00000100:00080000:3.0:1787696278.163095:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696278.163099:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696278.163105:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696278.163119:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134216960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696278.163129:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696278.163133:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696278.163135:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696278.163139:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4179 00000100:00080000:3.0:1787696278.163146:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696278.163158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696278.163164:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696278.163176:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134217088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696278.163187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696278.163193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696278.163197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696278.163212:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4179 00000100:00080000:3.0:1787696278.163221:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696278.163229:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696278.163235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696278.163253:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134217216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696278.163264:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696278.163268:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696278.163270:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696278.163275:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4179 00000100:00080000:3.0:1787696278.163280:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696278.163290:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696278.163296:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696278.163314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134217344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696278.163330:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696278.163335:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696278.163339:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696278.163348:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4179 00000100:00080000:3.0:1787696278.163354:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696278.163362:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696278.163369:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696278.163382:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134217472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696278.739059:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696279.763055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696280.787061:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696282.835012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696283.282967:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696283.282978:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696283.282982:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696283.282993:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4185 00000100:00080000:3.0:1787696283.283005:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696283.283013:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696283.283053:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134217600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696283.283079:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696283.283082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696283.283084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696283.283087:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4185 00000100:00080000:3.0:1787696283.283090:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696283.283093:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696283.283100:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696283.283109:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134217728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696283.283118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696283.283120:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696283.283121:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696283.283126:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4185 00000100:00080000:3.0:1787696283.283129:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696283.283134:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696283.283138:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696283.283145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134217856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696283.283151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696283.283154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696283.283156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696283.283159:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4185 00000100:00080000:3.0:1787696283.283161:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696283.283163:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696283.283166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696283.283175:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134217984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696283.283180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696283.283182:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696283.283184:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696283.283187:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4185 00000100:00080000:3.0:1787696283.283190:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696283.283193:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696283.283196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696283.283206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134218112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696283.283226:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696283.283229:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696283.283230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696283.283234:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4185 00000100:00080000:3.0:1787696283.283239:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696283.283246:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696283.283253:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696283.283265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134218240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696283.283296:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696283.283304:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696283.283308:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696283.283316:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4185 00000100:00080000:3.0:1787696283.283319:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696283.283326:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696283.283329:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696283.283335:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134218368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696283.859021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696284.883034:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696285.907051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696286.931081:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696288.402997:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696288.403007:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696288.403012:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696288.403023:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4190 00000100:00080000:2.0:1787696288.403033:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696288.403041:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696288.403080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134218496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696288.403117:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696288.403121:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696288.403123:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696288.403128:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4190 00000100:00080000:2.0:1787696288.403132:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696288.403136:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696288.403146:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696288.403160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134218624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696288.403174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696288.403181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696288.403186:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696288.403191:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4190 00000100:00080000:2.0:1787696288.403195:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696288.403200:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696288.403206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696288.403220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134218752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696288.403231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696288.403235:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696288.403240:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696288.403247:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4190 00000100:00080000:2.0:1787696288.403252:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696288.403257:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696288.403263:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696288.403277:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134218880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696288.403289:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696288.403299:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696288.403304:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696288.403313:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4190 00000100:00080000:2.0:1787696288.403320:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696288.403330:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696288.403348:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696288.403368:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134219008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696288.403382:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696288.403389:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696288.403393:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696288.403400:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4190 00000100:00080000:2.0:1787696288.403408:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696288.403417:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696288.403428:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696288.403449:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134219136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696288.403476:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696288.403484:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696288.403488:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696288.403495:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4190 00000100:00080000:2.0:1787696288.403504:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696288.403508:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696288.403524:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696288.403546:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134219264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696288.979066:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696290.003026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696291.027045:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696292.051026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696293.075009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696293.523125:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134219392:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696293.523199:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696293.523252:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696293/real 1787696293] req@ffff9d85b9d2f480 x1874531134219392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696293.523282:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134219392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696293.523292:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696293.523295:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696293.523299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134219392:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696293.523305:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134219520:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696293.523324:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696293.523333:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696293/real 1787696293] req@ffff9d85b9d2c000 x1874531134219520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696293.523341:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134219520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696293.523347:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696293.523349:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696293.523351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134219520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696293.523369:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134219648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696293.523401:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696293.523410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696293/real 1787696293] req@ffff9d85b9d2ce00 x1874531134219648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696293.523423:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134219648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696293.523428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696293.523430:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696293.523439:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134219648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696293.523444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134219776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696293.523456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696293.523462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696293/real 1787696293] req@ffff9d85b9d2e300 x1874531134219776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696293.523470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134219776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696293.523475:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696293.523477:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696293.523479:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134219776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696293.523482:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134219904:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696293.523492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696293.523497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696293/real 1787696293] req@ffff9d85b9d2fb80 x1874531134219904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696293.523504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134219904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696293.523514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696293.523516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696293.523518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134219904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696293.523521:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696293.523530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696293.523536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696293/real 1787696293] req@ffff9d85b9d2ca80 x1874531134220032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696293.523542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134220032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696293.523547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696293.523549:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696293.523551:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696293.523554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696293.523563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696293.523568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696293/real 1787696293] req@ffff9d85b9d2e680 x1874531134220160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696293.523576:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134220160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696293.523580:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696293.523582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696293.523585:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220160:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696295.122970:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696296.147024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696297.170991:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696298.194992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696298.642974:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696298.642984:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696298.642989:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696298.643000:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4200 00000100:00080000:0.0:1787696298.643011:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696298.643019:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696298.643057:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134220288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696298.643087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696298.643092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696298.643094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696298.643098:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4200 00000100:00080000:0.0:1787696298.643102:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696298.643107:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696298.643117:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696298.643132:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134220416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696298.643142:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134220288:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787696298.643163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696298.643167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696298.643170:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696298.643175:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4200 00000100:00080000:0.0:1787696298.643179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696298.643184:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696298.643190:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696298.643203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134220544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696298.643221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696298.643226:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787696298.643228:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787696298.643230:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696298.643237:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4200 00000100:00080000:0.0:1787696298.643242:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696298.643247:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696298.643252:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696298.643268:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134220672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696298.643277:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696298/real 1787696298] req@ffff9d85b9d09180 x1874531134220288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696298.643281:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696298.643303:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696298.643308:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696298.643314:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4200 00000100:00100000:3.0:1787696298.643318:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134220288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696298.643321:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696298.643332:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787696298.643332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696298.643341:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:0.0:1787696298.643342:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696298.643345:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134220288:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787696298.643360:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134220800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696298.643369:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134220416:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787696298.643377:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696298.643383:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696298.643385:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696298.643391:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4200 00000800:00000100:3.0:1787696298.643396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787696298.643400:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696298.643407:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696298.643411:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696298/real 1787696298] req@ffff9d85b9d0a300 x1874531134220416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787696298.643415:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696298.643426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134220416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696298.643434:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787696298.643435:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134220928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696298.643441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696298.643449:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134220416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696298.643460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134220544:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787696298.643461:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696298.643470:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696298.643473:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696298.643479:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4200 00000800:00000100:3.0:1787696298.643485:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787696298.643488:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696298.643495:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696298/real 1787696298] req@ffff9d85b9d09500 x1874531134220544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696298.643496:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696298.643506:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696298.643511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134220544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696298.643518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696298.643521:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787696298.643526:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134221056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696298.643527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134220544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696298.643539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134220672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696298.643567:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696298.643576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696298/real 1787696298] req@ffff9d85b9d09f80 x1874531134220672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696298.643585:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134220672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696298.643590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696298.643592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696298.643594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134220672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696298.643604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220800:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696298.643615:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696298.643621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696298/real 1787696298] req@ffff9d85b9d09c00 x1874531134220800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696298.643633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134220800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696298.643638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696298.643640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696298.643642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696298.643645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696298.643655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696298.643661:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696298/real 1787696298] req@ffff9d85b9d0b800 x1874531134220928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696298.643673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134220928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696298.643681:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696298.643684:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696298.643687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134220928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696298.643693:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696298.643722:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696298.643731:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696298/real 1787696298] req@ffff9d85b9d08700 x1874531134221056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696298.643740:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134221056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696298.643755:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696298.643758:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696298.643771:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221056:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696299.219037:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696301.267008:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696302.291016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696303.314974:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696303.762980:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696303.762989:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696303.763000:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696303.763012:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4205 00000100:00080000:0.0:1787696303.763023:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696303.763030:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696303.763074:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134221184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696303.763113:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696303.763123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696303.763125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696303.763150:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4205 00000100:00080000:0.0:1787696303.763155:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696303.763160:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696303.763169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696303.763180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134221312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696303.763188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696303.763191:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696303.763192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696303.763194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4205 00000100:00080000:0.0:1787696303.763210:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696303.763213:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696303.763217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696303.763225:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134221440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696303.763232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696303.763234:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696303.763235:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696303.763237:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4205 00000100:00080000:0.0:1787696303.763245:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696303.763247:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696303.763253:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696303.763260:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134221568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696303.763266:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696303.763269:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696303.763270:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696303.763271:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4205 00000100:00080000:0.0:1787696303.763273:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696303.763276:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696303.763279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696303.763288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134221696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696303.763297:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696303.763301:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696303.763302:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696303.763306:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4205 00000100:00080000:0.0:1787696303.763309:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696303.763313:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696303.763318:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696303.763328:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134221824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696303.763338:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696303.763342:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696303.763344:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696303.763348:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4205 00000100:00080000:0.0:1787696303.763351:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696303.763355:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696303.763359:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696303.763370:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134221952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696303.763409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134221184:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696303.763470:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696303.763525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696303/real 1787696303] req@ffff9d85b9d08700 x1874531134221184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696303.763545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134221184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696303.763557:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696303.763561:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696303.763565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134221184:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696303.763573:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134221312:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696303.763596:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696303.763610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696303/real 1787696303] req@ffff9d85b9d0b800 x1874531134221312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696303.763623:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134221312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696303.763632:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696303.763636:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696303.763640:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134221312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696303.763646:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134221440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696303.763666:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696303.763677:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696303/real 1787696303] req@ffff9d85b9d09c00 x1874531134221440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696303.763689:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134221440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696303.763697:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696303.763700:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696303.763704:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134221440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696303.763710:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134221568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696303.763730:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696303.763740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696303/real 1787696303] req@ffff9d85b9d09f80 x1874531134221568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696303.763753:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134221568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696303.763761:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696303.763764:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696303.763768:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134221568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696303.763774:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221696:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696303.763793:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696303.763804:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696303/real 1787696303] req@ffff9d85b9d09500 x1874531134221696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696303.763817:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134221696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696303.763826:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696303.763829:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696303.763832:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696303.763839:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696303.763887:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696303.763900:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696303/real 1787696303] req@ffff9d85b9d0a300 x1874531134221824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696303.763912:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134221824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696303.763921:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696303.763925:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696303.763928:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696303.763934:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696303.763954:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696303.763965:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696303/real 1787696303] req@ffff9d85b9d09180 x1874531134221952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696303.763977:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134221952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696303.763987:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696303.763990:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696303.763994:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134221952:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696304.339076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696305.363010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696307.410997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696308.434992:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696308.883009:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696308.883019:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696308.883024:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696308.883035:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4210 00000100:00080000:3.0:1787696308.883047:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696308.883055:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696308.883093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134222080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696308.883135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696308.883140:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696308.883142:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696308.883147:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4210 00000100:00080000:3.0:1787696308.883151:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696308.883156:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696308.883165:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696308.883181:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134222208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696308.883199:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696308.883205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696308.883207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696308.883211:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4210 00000100:00080000:3.0:1787696308.883215:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696308.883220:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696308.883226:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696308.883239:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134222336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696308.883255:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696308.883260:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696308.883262:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696308.883268:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4210 00000100:00080000:3.0:1787696308.883272:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696308.883276:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696308.883281:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696308.883291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134222464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696308.883301:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696308.883305:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696308.883309:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696308.883319:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4210 00000100:00080000:3.0:1787696308.883327:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696308.883333:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696308.883340:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696308.883359:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134222592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696308.883378:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696308.883386:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696308.883389:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696308.883394:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4210 00000100:00080000:3.0:1787696308.883397:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696308.883401:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696308.883406:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696308.883422:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134222720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696308.883434:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696308.883440:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696308.883444:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696308.883450:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4210 00000100:00080000:3.0:1787696308.883455:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696308.883462:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696308.883472:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696308.883489:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134222848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696309.458990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696310.482980:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696311.507029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696313.555010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696314.003035:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696314.003048:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696314.003053:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696314.003064:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4215 00000100:00080000:2.0:1787696314.003076:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696314.003085:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696314.003134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134222976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696314.003172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696314.003176:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696314.003177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696314.003182:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4215 00000100:00080000:2.0:1787696314.003185:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696314.003189:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696314.003200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696314.003212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134223104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696314.003228:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696314.003235:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696314.003238:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696314.003242:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4215 00000100:00080000:2.0:1787696314.003245:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696314.003248:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696314.003252:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696314.003261:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134223232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696314.003273:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696314.003276:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696314.003278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696314.003281:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4215 00000100:00080000:2.0:1787696314.003284:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696314.003286:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696314.003289:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696314.003299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134223360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696314.003306:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696314.003308:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696314.003310:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696314.003312:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4215 00000100:00080000:2.0:1787696314.003315:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696314.003318:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696314.003322:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696314.003330:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134223488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696314.003337:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696314.003340:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696314.003342:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696314.003344:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4215 00000100:00080000:2.0:1787696314.003346:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696314.003348:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696314.003352:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696314.003367:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134223616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696314.003378:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696314.003382:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696314.003383:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696314.003387:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4215 00000100:00080000:2.0:1787696314.003390:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696314.003393:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696314.003397:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696314.003409:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134223744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696314.579056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696315.603071:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696316.627055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696317.075236:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696314/real 1787696317] req@ffff9d85babd6a00 x1874531134223104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696317.075268:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134223104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696317.075278:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696317.075283:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696317.075287:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134223104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696317.075294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696314/real 1787696317] req@ffff9d85babd6680 x1874531134223232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696317.075307:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134223232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696317.075314:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696317.075317:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696317.075320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134223232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696317.075332:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696314/real 1787696317] req@ffff9d85babd4000 x1874531134223360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696317.075343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134223360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696317.075351:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696317.075354:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696317.075358:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134223360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696317.075364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696314/real 1787696317] req@ffff9d85babd4380 x1874531134223488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696317.075377:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134223488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696317.075384:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696317.075387:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696317.075390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134223488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696317.075396:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696314/real 1787696317] req@ffff9d85babd6300 x1874531134222976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696317.075407:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134222976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696317.075413:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696317.075416:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696317.075419:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134222976:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696317.075444:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696314/real 1787696317] req@ffff9d85babd7480 x1874531134223616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696317.075455:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134223616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696317.075463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696317.075467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696317.075470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134223616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696317.075475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696314/real 1787696317] req@ffff9d85babd5c00 x1874531134223744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696317.075487:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134223744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696317.075494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696317.075496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696317.075499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134223744:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696317.651030:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696317.651202:0:2361:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9d85b9d2df80 x1874531134223872/t0(0) o103->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1787696318.674982:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696319.122968:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696319.122978:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696319.122981:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696319.122989:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4220 00000100:00080000:3.0:1787696319.122998:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696319.123003:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696319.123029:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134224000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696319.123051:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696319.123054:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696319.123055:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696319.123058:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4220 00000100:00080000:3.0:1787696319.123060:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696319.123062:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696319.123069:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696319.123076:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134224128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696319.123091:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696319.123094:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696319.123096:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696319.123100:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4220 00000100:00080000:3.0:1787696319.123104:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696319.123110:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696319.123118:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696319.123132:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134224256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696319.123141:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696319.123145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696319.123147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696319.123150:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4220 00000100:00080000:3.0:1787696319.123154:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696319.123158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696319.123162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696319.123175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134224384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696319.123192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696319.123197:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696319.123199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696319.123204:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4220 00000100:00080000:3.0:1787696319.123209:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696319.123214:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696319.123223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696319.123237:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134224512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696319.123246:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696319.123254:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696319.123257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696319.123262:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4220 00000100:00080000:3.0:1787696319.123268:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696319.123273:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696319.123278:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696319.123292:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134224640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696319.123306:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696319.123311:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696319.123315:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696319.123321:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4220 00000100:00080000:3.0:1787696319.123326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696319.123333:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696319.123338:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696319.123350:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134224768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696319.699093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696320.723120:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696321.747053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696322.771009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696324.242911:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696324.242919:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696324.242923:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696324.242932:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4226 00000100:00080000:3.0:1787696324.242942:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696324.242948:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696324.242982:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134224896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696324.243009:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696324.243013:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696324.243014:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696324.243018:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4226 00000100:00080000:3.0:1787696324.243021:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696324.243024:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696324.243033:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696324.243044:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134225024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696324.243059:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696324.243063:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696324.243065:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696324.243069:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4226 00000100:00080000:3.0:1787696324.243072:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696324.243076:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696324.243081:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696324.243091:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134225152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696324.243099:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696324.243102:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696324.243105:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696324.243108:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4226 00000100:00080000:3.0:1787696324.243112:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696324.243119:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696324.243126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696324.243137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134225280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696324.243147:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696324.243150:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696324.243151:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696324.243154:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4226 00000100:00080000:3.0:1787696324.243158:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696324.243168:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696324.243175:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696324.243190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134225408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696324.243211:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696324.243216:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696324.243220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696324.243228:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4226 00000100:00080000:3.0:1787696324.243233:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696324.243237:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696324.243240:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696324.243252:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134225536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696324.243266:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696324.243270:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696324.243272:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696324.243276:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4226 00000100:00080000:3.0:1787696324.243279:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696324.243285:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696324.243291:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696324.243301:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134225664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696324.819022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696325.842996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696326.866976:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696327.890997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696328.915050:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696329.362957:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696329.362968:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696329.362972:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696329.362983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4231 00000100:00080000:0.0:1787696329.362994:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696329.363001:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696329.363045:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134225792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696329.363068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696329.363071:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696329.363072:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696329.363075:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4231 00000100:00080000:0.0:1787696329.363077:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696329.363079:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696329.363088:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696329.363096:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134225920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696329.363103:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696329.363106:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696329.363107:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696329.363109:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4231 00000100:00080000:0.0:1787696329.363111:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696329.363114:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696329.363117:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696329.363123:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134226048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696329.363130:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696329.363134:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696329.363135:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696329.363143:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4231 00000100:00080000:0.0:1787696329.363145:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696329.363148:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696329.363150:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696329.363157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134226176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696329.363163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696329.363166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696329.363167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696329.363170:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4231 00000100:00080000:0.0:1787696329.363172:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696329.363175:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696329.363178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696329.363184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134226304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696329.363189:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696329.363192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696329.363193:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696329.363195:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4231 00000100:00080000:0.0:1787696329.363212:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696329.363215:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696329.363217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696329.363224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134226432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696329.363235:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696329.363239:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696329.363241:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696329.363246:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4231 00000100:00080000:0.0:1787696329.363250:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696329.363254:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696329.363259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696329.363273:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134226560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696330.963063:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696331.986970:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696333.011023:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696334.034985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696334.482937:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696334.482944:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696334.482948:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696334.482958:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4236 00000100:00080000:3.0:1787696334.482968:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696334.482976:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696334.483013:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134226688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696334.483035:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696334.483038:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696334.483039:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696334.483043:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4236 00000100:00080000:3.0:1787696334.483046:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696334.483049:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696334.483056:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696334.483066:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134226816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696334.483074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696334.483077:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696334.483078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696334.483081:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4236 00000100:00080000:3.0:1787696334.483083:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696334.483087:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696334.483092:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696334.483116:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134226944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696334.483124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696334.483127:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696334.483129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696334.483133:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4236 00000100:00080000:3.0:1787696334.483135:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696334.483138:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696334.483142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696334.483153:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134227072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696334.483163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696334.483166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696334.483167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696334.483170:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4236 00000100:00080000:3.0:1787696334.483172:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696334.483175:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696334.483178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696334.483185:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134227200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696334.483192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696334.483195:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696334.483208:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696334.483217:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4236 00000100:00080000:3.0:1787696334.483222:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696334.483231:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696334.483239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696334.483257:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134227328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696334.483283:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696334.483288:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696334.483291:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696334.483297:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4236 00000100:00080000:3.0:1787696334.483302:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696334.483308:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696334.483317:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696334.483343:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134227456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696335.059033:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696337.107007:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696338.131065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696339.155035:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696339.602911:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696339.602920:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696339.602924:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696339.602932:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4241 00000100:00080000:3.0:1787696339.602941:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696339.602947:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696339.602980:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134227584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696339.603007:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696339.603010:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696339.603011:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696339.603014:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4241 00000100:00080000:3.0:1787696339.603017:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696339.603020:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696339.603027:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696339.603035:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134227712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696339.603042:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696339.603044:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696339.603045:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696339.603048:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4241 00000100:00080000:3.0:1787696339.603049:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696339.603051:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696339.603054:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696339.603059:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134227840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696339.603066:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696339.603069:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696339.603070:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696339.603073:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4241 00000100:00080000:3.0:1787696339.603075:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696339.603078:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696339.603081:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696339.603087:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134227968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696339.603108:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696339.603111:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696339.603112:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696339.603115:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4241 00000100:00080000:3.0:1787696339.603118:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696339.603121:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696339.603124:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696339.603133:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134228096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696339.603139:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696339.603142:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696339.603143:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696339.603145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4241 00000100:00080000:3.0:1787696339.603147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696339.603150:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696339.603154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696339.603167:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134228224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696339.603177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696339.603181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696339.603183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696339.603187:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4241 00000100:00080000:3.0:1787696339.603192:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696339.603197:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696339.603203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696339.603225:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134228352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696340.179013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696341.202990:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696343.251022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696344.275040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696344.722996:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696344.723012:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696344.723018:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696344.723032:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4246 00000100:00080000:3.0:1787696344.723047:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696344.723057:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696344.723108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134228480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696344.723147:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696344.723152:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696344.723161:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696344.723166:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4246 00000100:00080000:3.0:1787696344.723169:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696344.723174:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696344.723187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696344.723200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134228608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696344.723218:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696344.723224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696344.723226:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696344.723231:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4246 00000100:00080000:3.0:1787696344.723235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696344.723239:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696344.723246:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696344.723259:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134228736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696344.723270:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696344.723274:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696344.723276:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696344.723280:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4246 00000100:00080000:3.0:1787696344.723285:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696344.723290:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696344.723295:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696344.723313:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134228864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696344.723325:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696344.723329:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696344.723331:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696344.723334:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4246 00000100:00080000:3.0:1787696344.723337:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696344.723341:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696344.723347:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696344.723360:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134228992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696344.723374:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696344.723383:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696344.723385:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696344.723390:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4246 00000100:00080000:3.0:1787696344.723396:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696344.723406:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696344.723412:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696344.723428:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134229120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696344.723440:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696344.723443:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696344.723445:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696344.723451:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4246 00000100:00080000:3.0:1787696344.723459:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696344.723463:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696344.723469:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696344.723483:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134229248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696345.299128:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696346.322979:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696347.347026:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696349.395021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696349.842935:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696349.842946:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696349.842951:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696349.842962:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4251 00000100:00080000:3.0:1787696349.842974:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696349.842982:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696349.843026:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134229376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696349.843059:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696349.843064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696349.843066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696349.843071:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4251 00000100:00080000:3.0:1787696349.843076:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696349.843081:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696349.843094:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696349.843108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134229504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696349.843119:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696349.843121:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696349.843123:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696349.843129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4251 00000100:00080000:3.0:1787696349.843133:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696349.843140:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696349.843146:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696349.843158:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134229632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696349.843172:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696349.843174:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696349.843176:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696349.843181:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4251 00000100:00080000:3.0:1787696349.843183:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696349.843186:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696349.843189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696349.843197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134229760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696349.843203:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696349.843206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696349.843209:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696349.843212:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4251 00000100:00080000:3.0:1787696349.843214:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696349.843217:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696349.843220:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696349.843228:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134229888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696349.843249:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696349.843253:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696349.843255:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696349.843259:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4251 00000100:00080000:3.0:1787696349.843263:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696349.843267:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696349.843271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696349.843281:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134230016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696349.843288:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696349.843290:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696349.843291:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696349.843296:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4251 00000100:00080000:3.0:1787696349.843303:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696349.843317:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696349.843333:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696349.843344:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134230144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696350.419022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696351.442974:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696352.467037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696353.491022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696354.962973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696354.962981:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696354.962985:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696354.962993:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4256 00000100:00080000:3.0:1787696354.963002:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696354.963009:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696354.963044:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134230272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696354.963078:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696354.963083:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696354.963085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696354.963090:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4256 00000100:00080000:3.0:1787696354.963094:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696354.963098:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696354.963121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696354.963139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134230400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696354.963153:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696354.963156:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696354.963158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696354.963163:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4256 00000100:00080000:3.0:1787696354.963168:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696354.963172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696354.963178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696354.963191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134230528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696354.963205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696354.963210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696354.963212:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696354.963218:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4256 00000100:00080000:3.0:1787696354.963224:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696354.963229:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696354.963235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696354.963250:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134230656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696354.963271:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696354.963277:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696354.963280:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696354.963285:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4256 00000100:00080000:3.0:1787696354.963291:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696354.963302:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696354.963310:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696354.963341:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134230784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696354.963360:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696354.963364:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696354.963365:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696354.963371:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4256 00000100:00080000:3.0:1787696354.963377:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696354.963386:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696354.963392:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696354.963406:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134230912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696354.963420:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696354.963423:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696354.963426:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696354.963435:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4256 00000100:00080000:3.0:1787696354.963441:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696354.963449:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696354.963457:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696354.963469:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134231040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696355.539010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696356.563116:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696357.587016:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696358.611027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696359.635014:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696360.082985:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696360.083000:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696360.083005:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696360.083019:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4261 00000100:00080000:3.0:1787696360.083035:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696360.083043:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696360.083097:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134231168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696360.083124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696360.083127:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696360.083129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696360.083132:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4261 00000100:00080000:3.0:1787696360.083135:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696360.083137:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696360.083147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696360.083156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134231296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696360.083165:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696360.083168:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696360.083169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696360.083173:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4261 00000100:00080000:3.0:1787696360.083174:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696360.083177:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696360.083180:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696360.083187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134231424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696360.083197:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696360.083204:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696360.083207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696360.083211:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4261 00000100:00080000:3.0:1787696360.083213:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696360.083215:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696360.083218:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696360.083259:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134231552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696360.083271:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696360.083276:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696360.083278:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696360.083282:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4261 00000100:00080000:3.0:1787696360.083287:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696360.083292:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696360.083296:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696360.083308:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134231680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696360.083316:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696360.083320:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696360.083321:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696360.083325:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4261 00000100:00080000:3.0:1787696360.083328:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696360.083331:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696360.083334:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696360.083341:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134231808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696360.083346:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696360.083349:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696360.083350:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696360.083353:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4261 00000100:00080000:3.0:1787696360.083356:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696360.083358:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696360.083361:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696360.083367:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134231936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696361.683034:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696362.707093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696363.731034:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696364.755034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696365.202972:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696365.202983:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696365.202987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696365.202999:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4266 00000100:00080000:3.0:1787696365.203012:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696365.203021:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696365.203068:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134232064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696365.203098:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696365.203101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696365.203102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696365.203105:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4266 00000100:00080000:3.0:1787696365.203108:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696365.203111:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696365.203121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696365.203131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134232192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696365.203154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696365.203157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696365.203158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696365.203161:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4266 00000100:00080000:3.0:1787696365.203164:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696365.203170:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696365.203178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696365.203187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134232320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696365.203193:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696365.203195:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696365.203197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696365.203199:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4266 00000100:00080000:3.0:1787696365.203201:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696365.203203:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696365.203206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696365.203216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134232448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696365.203222:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696365.203224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696365.203225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696365.203228:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4266 00000100:00080000:3.0:1787696365.203231:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696365.203234:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696365.203237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696365.203243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134232576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696365.203248:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696365.203252:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696365.203253:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696365.203260:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4266 00000100:00080000:3.0:1787696365.203264:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696365.203266:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696365.203270:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696365.203277:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134232704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696365.203284:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696365.203287:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696365.203288:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696365.203291:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4266 00000100:00080000:3.0:1787696365.203293:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696365.203296:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696365.203299:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696365.203307:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134232832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696365.779043:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696367.826974:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696368.851019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696369.875135:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696370.322986:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696370.323000:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696370.323006:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696370.323021:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4272 00000100:00080000:3.0:1787696370.323035:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696370.323046:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696370.323119:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134232960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696370.323157:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696370.323162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696370.323164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696370.323169:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4272 00000100:00080000:3.0:1787696370.323174:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696370.323180:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696370.323193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696370.323212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134233088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696370.323222:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696370.323225:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696370.323227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696370.323232:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4272 00000100:00080000:3.0:1787696370.323237:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696370.323241:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696370.323248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696370.323263:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134233216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696370.323275:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696370.323279:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696370.323283:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696370.323291:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4272 00000100:00080000:3.0:1787696370.323295:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696370.323300:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696370.323305:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696370.323317:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134233344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696370.323324:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696370.323327:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696370.323328:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696370.323330:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4272 00000100:00080000:3.0:1787696370.323333:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696370.323337:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696370.323340:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696370.323352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134233472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696370.323362:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696370.323365:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696370.323369:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696370.323378:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4272 00000100:00080000:3.0:1787696370.323385:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696370.323390:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696370.323400:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696370.323419:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134233600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696370.323433:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696370.323439:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696370.323444:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696370.323450:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4272 00000100:00080000:3.0:1787696370.323455:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696370.323461:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696370.323468:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696370.323483:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134233728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696370.899024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696371.923099:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696373.971038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696374.994992:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696375.442984:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696375.442993:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696375.442997:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696375.443005:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4277 00000100:00080000:3.0:1787696375.443015:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696375.443021:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696375.443053:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134233856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696375.443078:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696375.443081:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696375.443083:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696375.443088:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4277 00000100:00080000:3.0:1787696375.443092:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696375.443097:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696375.443107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696375.443126:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134233984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696375.443143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696375.443147:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696375.443149:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696375.443154:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4277 00000100:00080000:3.0:1787696375.443158:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696375.443162:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696375.443169:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696375.443183:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134234112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696375.443205:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696375.443211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696375.443212:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696375.443216:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4277 00000100:00080000:3.0:1787696375.443220:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696375.443223:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696375.443228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696375.443240:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134234240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696375.443260:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696375.443263:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696375.443264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696375.443269:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4277 00000100:00080000:3.0:1787696375.443272:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696375.443279:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696375.443285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696375.443298:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134234368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696375.443314:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696375.443318:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696375.443319:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696375.443322:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4277 00000100:00080000:3.0:1787696375.443324:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696375.443327:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696375.443329:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696375.443340:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134234496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696375.443350:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696375.443352:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696375.443354:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696375.443357:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4277 00000100:00080000:3.0:1787696375.443360:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696375.443365:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696375.443372:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696375.443385:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134234624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696376.019052:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696377.043034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696378.067043:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696380.115035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696380.563123:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134234752:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696380.563223:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134234880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696380.563273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134235008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696380.563307:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134235136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696380.563328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134235264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696380.563354:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134235392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696380.563382:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134235520:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696381.139038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696382.163016:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696383.186979:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696383.635125:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696383.635163:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696383.635181:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696383.635229:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696383.635239:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696383.635248:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696383.635256:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696383.635264:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696383.635274:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787696383.635302:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696380/real 1787696383] req@ffff9d85b9d2e680 x1874531134235520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696383.635334:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134235520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696383.635346:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696383.635350:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696383.635355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134235520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696383.635362:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696380/real 1787696383] req@ffff9d85b9d2dc00 x1874531134235136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696383.635395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134235136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696383.635405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696383.635408:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696383.635412:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134235136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696383.635428:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696380/real 1787696383] req@ffff9d85b9d2f800 x1874531134235264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696383.635442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134235264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696383.635451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696383.635454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696383.635458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134235264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696383.635465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696380/real 1787696383] req@ffff9d85b9d2d500 x1874531134235392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696383.635477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134235392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696383.635485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696383.635488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696383.635492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134235392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696383.635499:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696380/real 1787696383] req@ffff9d85b9d2ed80 x1874531134234880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696383.635512:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134234880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696383.635521:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696383.635525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696383.635528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134234880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696383.635536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696380/real 1787696383] req@ffff9d85b9d2c700 x1874531134235008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696383.635550:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134235008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696383.635558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696383.635562:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696383.635565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134235008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696383.635574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696380/real 1787696383] req@ffff9d85b9d2c380 x1874531134234752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696383.635588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134234752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696383.635596:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696383.635600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696383.635602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134234752:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787696384.211003:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696385.682951:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696385.682960:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696385.682962:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696385.682970:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4287 00000100:00080000:3.0:1787696385.682986:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696385.682992:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696385.683020:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134235648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696385.683043:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696385.683047:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696385.683049:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696385.683055:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4287 00000100:00080000:3.0:1787696385.683060:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696385.683065:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696385.683074:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696385.683093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134235776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696385.683105:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696385.683109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696385.683111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696385.683115:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4287 00000100:00080000:3.0:1787696385.683120:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696385.683123:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696385.683139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696385.683157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134235904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696385.683169:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696385.683172:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696385.683174:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696385.683178:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4287 00000100:00080000:3.0:1787696385.683182:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696385.683186:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696385.683191:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696385.683207:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134236032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696385.683222:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696385.683228:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696385.683232:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696385.683241:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4287 00000100:00080000:3.0:1787696385.683248:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696385.683266:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696385.683273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696385.683287:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134236160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696385.683310:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696385.683315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696385.683317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696385.683323:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4287 00000100:00080000:3.0:1787696385.683331:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696385.683339:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696385.683345:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696385.683361:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134236288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696385.683380:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696385.683389:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696385.683394:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696385.683403:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4287 00000100:00080000:3.0:1787696385.683411:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696385.683418:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696385.683426:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696385.683443:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134236416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696386.259004:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696387.283030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696388.306976:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696389.330991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696390.355082:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696390.802952:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696390.802961:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696390.802964:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696390.802973:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4292 00000100:00080000:0.0:1787696390.802983:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696390.802989:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696390.803021:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134236544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696390.803043:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696390.803046:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696390.803047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696390.803050:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4292 00000100:00080000:0.0:1787696390.803053:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696390.803055:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696390.803062:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696390.803073:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134236672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696390.803080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696390.803082:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696390.803084:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696390.803087:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4292 00000100:00080000:0.0:1787696390.803090:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696390.803093:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696390.803098:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696390.803104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134236800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696390.803111:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696390.803122:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696390.803124:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696390.803130:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4292 00000100:00080000:0.0:1787696390.803133:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696390.803146:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696390.803150:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696390.803162:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134236928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696390.803169:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696390.803171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696390.803173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696390.803175:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4292 00000100:00080000:0.0:1787696390.803178:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696390.803180:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696390.803183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696390.803190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134237056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696390.803196:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696390.803199:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696390.803201:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696390.803204:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4292 00000100:00080000:0.0:1787696390.803221:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696390.803224:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696390.803227:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696390.803236:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134237184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696390.803252:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696390.803255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696390.803257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696390.803260:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4292 00000100:00080000:0.0:1787696390.803263:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696390.803266:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696390.803270:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696390.803279:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134237312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696392.402997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696393.427121:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696394.451003:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696395.475018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696395.923202:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134237440:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696395.923281:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696395.923347:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696395/real 1787696395] req@ffff9d85b9d2ca80 x1874531134237440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696395.923371:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134237440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696395.923395:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696395.923401:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696395.923405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134237440:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696395.923440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134237568:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696395.923468:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696395.923480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696395/real 1787696395] req@ffff9d85b9d2fb80 x1874531134237568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696395.923489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134237568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696395.923495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696395.923497:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696395.923500:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134237568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696395.923505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134237696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696395.923518:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696395.923525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696395/real 1787696395] req@ffff9d85b9d2e300 x1874531134237696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696395.923533:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134237696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696395.923538:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696395.923539:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696395.923541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134237696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696395.923545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134237824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696395.923557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696395.923562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696395/real 1787696395] req@ffff9d85b9d2ce00 x1874531134237824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696395.923570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134237824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696395.923575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696395.923577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696395.923579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134237824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696395.923582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134237952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696395.923592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696395.923597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696395/real 1787696395] req@ffff9d85b9d2c000 x1874531134237952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696395.923604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134237952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696395.923625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696395.923628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696395.923630:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134237952:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696395.923635:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134238080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696395.923650:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696395.923658:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696395/real 1787696395] req@ffff9d85b9d2f480 x1874531134238080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696395.923671:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134238080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696395.923676:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696395.923678:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696395.923680:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134238080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696395.923683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134238208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696395.923694:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696395.923700:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696395/real 1787696395] req@ffff9d85b9d2c380 x1874531134238208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696395.923707:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134238208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696395.923713:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696395.923714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696395.923716:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134238208:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696396.499047:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696398.547078:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696399.571046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696400.595031:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696401.042946:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696401.042954:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696401.042959:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696401.042967:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4302 00000100:00080000:3.0:1787696401.042981:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696401.042987:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696401.043019:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134238336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696401.043047:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696401.043049:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696401.043051:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696401.043054:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4302 00000100:00080000:3.0:1787696401.043057:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696401.043059:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696401.043067:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696401.043077:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134238464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696401.043087:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696401.043091:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696401.043096:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696401.043105:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4302 00000100:00080000:3.0:1787696401.043123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696401.043130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696401.043137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696401.043152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134238592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696401.043160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696401.043163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696401.043164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696401.043168:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4302 00000100:00080000:3.0:1787696401.043172:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696401.043175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696401.043179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696401.043199:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134238720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696401.043214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696401.043219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696401.043222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696401.043228:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4302 00000100:00080000:3.0:1787696401.043233:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696401.043238:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696401.043244:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696401.043265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134238848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696401.043288:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696401.043296:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696401.043300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696401.043316:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4302 00000100:00080000:3.0:1787696401.043321:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696401.043345:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696401.043362:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696401.043376:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134238976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696401.043404:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696401.043409:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696401.043412:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696401.043419:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4302 00000100:00080000:3.0:1787696401.043426:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696401.043438:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696401.043444:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696401.043462:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134239104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696401.619030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696402.643029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696404.691004:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696405.715022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696406.162929:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696406.162938:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696406.162942:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696406.162952:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4307 00000100:00080000:3.0:1787696406.162962:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696406.162971:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696406.163008:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134239232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696406.163032:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696406.163034:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696406.163036:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696406.163039:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4307 00000100:00080000:3.0:1787696406.163042:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696406.163046:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696406.163054:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696406.163064:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134239360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696406.163072:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696406.163076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696406.163078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696406.163083:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4307 00000100:00080000:3.0:1787696406.163088:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696406.163091:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696406.163094:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696406.163101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134239488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696406.163108:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696406.163110:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696406.163111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696406.163114:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4307 00000100:00080000:3.0:1787696406.163116:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696406.163118:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696406.163121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696406.163128:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134239616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696406.163134:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696406.163136:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696406.163147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696406.163155:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4307 00000100:00080000:3.0:1787696406.163158:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696406.163161:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696406.163165:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696406.163174:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134239744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696406.163180:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696406.163183:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696406.163184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696406.163186:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4307 00000100:00080000:3.0:1787696406.163189:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696406.163196:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696406.163199:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696406.163209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134239872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696406.163232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696406.163234:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696406.163236:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696406.163248:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4307 00000100:00080000:3.0:1787696406.163259:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696406.163267:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696406.163279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696406.163295:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134240000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696406.739013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696407.762990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696408.786969:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696410.835005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696411.282994:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696411.283002:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696411.283010:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696411.283018:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4313 00000100:00080000:0.0:1787696411.283027:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696411.283034:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696411.283080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134240128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696411.283113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696411.283117:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696411.283118:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696411.283121:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4313 00000100:00080000:0.0:1787696411.283125:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696411.283128:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696411.283137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696411.283153:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134240256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696411.283172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696411.283176:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696411.283179:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696411.283184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4313 00000100:00080000:0.0:1787696411.283189:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696411.283192:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696411.283197:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696411.283212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134240384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696411.283224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696411.283227:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696411.283229:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696411.283237:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4313 00000100:00080000:0.0:1787696411.283246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696411.283251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696411.283256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696411.283270:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134240512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696411.283292:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696411.283297:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696411.283301:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696411.283311:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4313 00000100:00080000:0.0:1787696411.283316:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696411.283325:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696411.283333:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696411.283372:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134240640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696411.283387:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696411.283389:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696411.283391:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696411.283394:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4313 00000100:00080000:0.0:1787696411.283405:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696411.283413:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696411.283419:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696411.283431:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134240768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696411.283462:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696411.283466:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696411.283467:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696411.283470:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4313 00000100:00080000:0.0:1787696411.283472:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696411.283477:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696411.283480:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696411.283493:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134240896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696411.859018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696412.883122:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696413.907025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696414.931011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696416.402975:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696416.402991:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696416.402994:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696416.403002:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4318 00000100:00080000:3.0:1787696416.403012:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696416.403018:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696416.403050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134241024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696416.403072:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696416.403075:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696416.403077:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696416.403080:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4318 00000100:00080000:3.0:1787696416.403082:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696416.403084:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696416.403092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696416.403105:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134241152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696416.403113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696416.403116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696416.403117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696416.403122:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4318 00000100:00080000:3.0:1787696416.403125:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696416.403130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696416.403138:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696416.403146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134241280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696416.403153:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696416.403155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696416.403156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696416.403158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4318 00000100:00080000:3.0:1787696416.403160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696416.403162:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696416.403165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696416.403174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134241408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696416.403180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696416.403183:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696416.403184:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696416.403186:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4318 00000100:00080000:3.0:1787696416.403189:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696416.403191:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696416.403196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696416.403204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134241536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696416.403213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696416.403215:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696416.403217:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696416.403219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4318 00000100:00080000:3.0:1787696416.403222:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696416.403224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696416.403228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696416.403234:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134241664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696416.403239:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696416.403241:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696416.403243:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696416.403246:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4318 00000100:00080000:3.0:1787696416.403249:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696416.403265:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696416.403269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696416.403283:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134241792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696416.979016:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696418.003057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696419.027027:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696420.051001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696421.075008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696421.523003:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696421.523016:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696421.523020:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696421.523032:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4323 00000100:00080000:3.0:1787696421.523045:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696421.523053:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696421.523101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134241920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696421.523131:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696421.523135:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696421.523136:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696421.523140:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4323 00000100:00080000:3.0:1787696421.523142:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696421.523144:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696421.523154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696421.523163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134242048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696421.523186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696421.523189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696421.523190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696421.523194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4323 00000100:00080000:3.0:1787696421.523198:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696421.523202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696421.523206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696421.523215:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134242176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696421.523223:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696421.523229:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696421.523233:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696421.523250:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4323 00000100:00080000:3.0:1787696421.523254:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696421.523260:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696421.523275:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696421.523292:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134242304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696421.523301:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696421.523312:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696421.523315:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696421.523320:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4323 00000100:00080000:3.0:1787696421.523325:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696421.523330:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696421.523335:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696421.523345:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134242432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696421.523355:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696421.523362:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696421.523368:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696421.523383:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4323 00000100:00080000:3.0:1787696421.523393:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696421.523398:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696421.523406:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696421.523422:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134242560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696421.523431:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696421.523441:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696421.523444:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696421.523452:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4323 00000100:00080000:3.0:1787696421.523460:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696421.523467:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696421.523480:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696421.523492:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134242688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696423.123040:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696424.147031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696425.170984:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696426.195001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696426.642916:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696426.642928:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696426.642933:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696426.642943:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4328 00000100:00080000:3.0:1787696426.642955:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696426.642963:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696426.643002:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134242816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696426.643033:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696426.643037:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696426.643039:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696426.643045:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4328 00000100:00080000:3.0:1787696426.643049:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696426.643054:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696426.643064:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696426.643093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134242944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696426.643106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696426.643112:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696426.643117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696426.643122:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4328 00000100:00080000:3.0:1787696426.643125:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696426.643130:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696426.643136:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696426.643150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134243072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696426.643160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696426.643163:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696426.643166:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696426.643170:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4328 00000100:00080000:3.0:1787696426.643174:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696426.643181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696426.643189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696426.643204:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134243200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696426.643215:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696426.643218:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696426.643220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696426.643224:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4328 00000100:00080000:3.0:1787696426.643229:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696426.643238:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696426.643247:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696426.643264:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134243328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696426.643289:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696426.643294:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696426.643297:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696426.643306:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4328 00000100:00080000:3.0:1787696426.643312:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696426.643317:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696426.643324:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696426.643343:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134243456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696426.643363:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696426.643372:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696426.643376:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696426.643386:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4328 00000100:00080000:3.0:1787696426.643393:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696426.643401:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696426.643410:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696426.643425:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134243584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696427.219018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696429.266993:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696430.291090:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696431.314995:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696431.762925:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696431.762935:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696431.762938:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696431.762947:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4333 00000100:00080000:3.0:1787696431.762957:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696431.762963:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696431.763027:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134243712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696431.763050:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696431.763053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696431.763054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696431.763057:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4333 00000100:00080000:3.0:1787696431.763060:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696431.763063:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696431.763071:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696431.763080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134243840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696431.763088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696431.763091:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696431.763092:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696431.763095:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4333 00000100:00080000:3.0:1787696431.763097:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696431.763100:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696431.763106:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696431.763115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134243968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696431.763122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696431.763124:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696431.763125:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696431.763128:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4333 00000100:00080000:3.0:1787696431.763131:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696431.763133:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696431.763136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696431.763146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134244096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696431.763152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696431.763155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696431.763156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696431.763158:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4333 00000100:00080000:3.0:1787696431.763160:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696431.763163:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696431.763167:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696431.763177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134244224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696431.763188:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696431.763191:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696431.763193:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696431.763196:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4333 00000100:00080000:3.0:1787696431.763198:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696431.763201:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696431.763204:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696431.763209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134244352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696431.763215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696431.763218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696431.763219:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696431.763221:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4333 00000100:00080000:3.0:1787696431.763223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696431.763224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696431.763227:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696431.763233:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134244480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696432.339049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696433.362986:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696435.411031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696436.435057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696436.882969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696436.882977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696436.882980:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696436.882988:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4338 00000100:00080000:3.0:1787696436.883004:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696436.883010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696436.883048:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134244608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696436.883079:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696436.883087:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696436.883088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696436.883092:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4338 00000100:00080000:3.0:1787696436.883095:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696436.883099:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696436.883107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696436.883124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134244736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696436.883152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696436.883155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696436.883156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696436.883161:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4338 00000100:00080000:3.0:1787696436.883164:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696436.883168:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696436.883173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696436.883185:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134244864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696436.883199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696436.883202:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696436.883203:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696436.883206:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4338 00000100:00080000:3.0:1787696436.883209:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696436.883212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696436.883216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696436.883224:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134244992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696436.883231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696436.883234:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696436.883235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696436.883237:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4338 00000100:00080000:3.0:1787696436.883240:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696436.883243:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696436.883246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696436.883257:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134245120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696436.883267:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696436.883272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696436.883274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696436.883283:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4338 00000100:00080000:3.0:1787696436.883290:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696436.883295:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696436.883300:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696436.883311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134245248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696436.883326:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696436.883335:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696436.883338:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696436.883341:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4338 00000100:00080000:3.0:1787696436.883345:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696436.883348:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696436.883352:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696436.883365:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134245376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696437.459045:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696438.483051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696439.507037:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696441.554981:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696442.002923:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696442.002931:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696442.002935:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696442.002943:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4343 00000100:00080000:3.0:1787696442.002953:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696442.002959:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696442.002991:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134245504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696442.003020:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696442.003023:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696442.003024:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696442.003027:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4343 00000100:00080000:3.0:1787696442.003029:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696442.003031:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696442.003040:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696442.003047:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134245632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696442.003055:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696442.003064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696442.003066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696442.003069:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4343 00000100:00080000:3.0:1787696442.003072:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696442.003075:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696442.003079:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696442.003088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134245760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696442.003095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696442.003098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696442.003099:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696442.003102:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4343 00000100:00080000:3.0:1787696442.003105:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696442.003107:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696442.003111:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696442.003119:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134245888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696442.003125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696442.003127:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696442.003129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696442.003131:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4343 00000100:00080000:3.0:1787696442.003134:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696442.003138:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696442.003144:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696442.003154:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134246016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696442.003160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696442.003162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696442.003163:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696442.003166:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4343 00000100:00080000:3.0:1787696442.003170:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696442.003172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696442.003175:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696442.003182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134246144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696442.003193:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696442.003197:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696442.003201:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696442.003209:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4343 00000100:00080000:3.0:1787696442.003215:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696442.003220:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696442.003223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696442.003238:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134246272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696442.579024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696443.602981:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696444.626962:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696445.650989:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696447.122911:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696447.122920:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696447.122923:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696447.122932:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4348 00000100:00080000:3.0:1787696447.122942:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696447.122949:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696447.122983:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134246400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696447.123014:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696447.123018:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696447.123020:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696447.123025:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4348 00000100:00080000:3.0:1787696447.123029:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696447.123032:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696447.123040:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696447.123052:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134246528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696447.123061:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696447.123063:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696447.123065:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696447.123071:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4348 00000100:00080000:3.0:1787696447.123076:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696447.123081:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696447.123085:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696447.123093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134246656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696447.123107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696447.123110:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696447.123111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696447.123115:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4348 00000100:00080000:3.0:1787696447.123118:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696447.123120:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696447.123123:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696447.123133:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134246784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696447.123144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696447.123146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696447.123147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696447.123150:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4348 00000100:00080000:3.0:1787696447.123152:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696447.123154:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696447.123157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696447.123162:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134246912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696447.123168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696447.123171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696447.123173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696447.123177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4348 00000100:00080000:3.0:1787696447.123181:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696447.123185:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696447.123193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696447.123211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134247040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696447.123224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696447.123230:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696447.123236:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696447.123243:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4348 00000100:00080000:3.0:1787696447.123250:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696447.123255:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696447.123261:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696447.123275:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134247168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696447.699017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696448.723000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696449.747070:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696450.195108:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696450.195145:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696450.195165:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696450.195201:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696450.195213:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696450.195224:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696450.195237:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696450.195247:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696450.195258:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:3.0:1787696450.771017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696451.795008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696452.242902:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696452.242910:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696452.242914:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696452.242922:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4354 00000100:00080000:3.0:1787696452.242931:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696452.242937:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696452.242966:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134247296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696452.242989:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696452.242992:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696452.242994:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696452.242997:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4354 00000100:00080000:3.0:1787696452.243000:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696452.243002:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696452.243009:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696452.243016:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134247424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696452.243028:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696452.243030:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696452.243032:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696452.243034:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4354 00000100:00080000:3.0:1787696452.243037:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696452.243039:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696452.243043:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696452.243050:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134247552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696452.243060:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696452.243064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696452.243065:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696452.243069:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4354 00000100:00080000:3.0:1787696452.243071:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696452.243075:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696452.243080:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696452.243092:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134247680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696452.243103:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696452.243106:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696452.243108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696452.243111:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4354 00000100:00080000:3.0:1787696452.243116:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696452.243120:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696452.243125:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696452.243142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134247808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696452.243153:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696452.243157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696452.243159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696452.243164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4354 00000100:00080000:3.0:1787696452.243168:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696452.243179:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696452.243185:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696452.243200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134247936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696452.243224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696452.243229:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696452.243232:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696452.243242:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4354 00000100:00080000:3.0:1787696452.243250:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696452.243258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696452.243268:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696452.243286:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134248064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696453.843009:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696454.866973:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696455.891008:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696456.915005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696457.362960:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696457.362968:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696457.362971:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696457.362979:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4359 00000100:00080000:3.0:1787696457.362988:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696457.362994:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696457.363035:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134248192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696457.363056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696457.363059:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696457.363061:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696457.363064:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4359 00000100:00080000:3.0:1787696457.363066:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696457.363072:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696457.363079:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696457.363090:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134248320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696457.363101:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696457.363104:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696457.363106:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696457.363111:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4359 00000100:00080000:3.0:1787696457.363118:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696457.363126:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696457.363133:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696457.363147:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134248448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696457.363156:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696457.363159:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696457.363161:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696457.363163:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4359 00000100:00080000:3.0:1787696457.363166:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696457.363169:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696457.363173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696457.363185:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134248576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696457.363198:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696457.363202:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696457.363205:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696457.363209:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4359 00000100:00080000:3.0:1787696457.363213:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696457.363218:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696457.363224:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696457.363239:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134248704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696457.363281:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696457.363284:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696457.363288:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696457.363294:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4359 00000100:00080000:3.0:1787696457.363300:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696457.363306:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696457.363314:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696457.363325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134248832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696457.363333:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696457.363335:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696457.363336:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696457.363339:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4359 00000100:00080000:3.0:1787696457.363342:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696457.363348:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696457.363353:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696457.363367:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134248960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696457.939037:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696459.987039:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696461.010993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696462.035026:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696462.482922:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696462.482930:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696462.482934:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696462.482942:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4364 00000100:00080000:3.0:1787696462.482952:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696462.482958:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696462.482990:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134249088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696462.483015:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696462.483018:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696462.483019:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696462.483022:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4364 00000100:00080000:3.0:1787696462.483025:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696462.483027:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696462.483034:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696462.483041:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134249216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696462.483048:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696462.483051:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696462.483053:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696462.483056:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4364 00000100:00080000:3.0:1787696462.483058:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696462.483061:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696462.483064:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696462.483070:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134249344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696462.483079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696462.483085:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696462.483086:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696462.483089:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4364 00000100:00080000:3.0:1787696462.483091:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696462.483093:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696462.483096:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696462.483104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134249472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696462.483109:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696462.483112:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696462.483113:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696462.483116:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4364 00000100:00080000:3.0:1787696462.483119:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696462.483121:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696462.483123:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696462.483130:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134249600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696462.483135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696462.483141:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696462.483144:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696462.483147:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4364 00000100:00080000:3.0:1787696462.483150:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696462.483152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696462.483156:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696462.483163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134249728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696462.483169:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696462.483171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696462.483171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696462.483174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4364 00000100:00080000:3.0:1787696462.483176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696462.483179:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696462.483183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696462.483194:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134249856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696463.059016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696464.082986:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696466.131003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696467.155011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696467.602940:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696467.602948:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696467.602952:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696467.602960:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4369 00000100:00080000:0.0:1787696467.602968:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696467.602974:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696467.603006:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134249984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696467.603027:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696467.603030:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696467.603031:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696467.603034:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4369 00000100:00080000:0.0:1787696467.603042:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696467.603046:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696467.603055:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696467.603065:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134250112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696467.603074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696467.603080:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696467.603084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696467.603087:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4369 00000100:00080000:0.0:1787696467.603090:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696467.603092:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696467.603096:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696467.603104:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134250240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696467.603111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696467.603114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696467.603116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696467.603118:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4369 00000100:00080000:0.0:1787696467.603128:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696467.603130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696467.603133:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696467.603143:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134250368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696467.603151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696467.603153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696467.603155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696467.603158:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4369 00000100:00080000:0.0:1787696467.603161:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696467.603164:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696467.603175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696467.603190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134250496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696467.603207:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696467.603213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696467.603217:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696467.603227:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4369 00000100:00080000:0.0:1787696467.603231:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696467.603237:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696467.603257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696467.603273:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134250624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696467.603289:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696467.603293:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696467.603296:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696467.603300:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4369 00000100:00080000:0.0:1787696467.603307:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696467.603314:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696467.603320:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696467.603337:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134250752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696468.179028:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696469.203062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696470.227066:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696472.275003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696472.722964:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696472.722975:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696472.722979:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696472.722990:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4374 00000100:00080000:3.0:1787696472.723004:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696472.723012:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696472.723058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134250880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696472.723087:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696472.723090:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696472.723092:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696472.723095:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4374 00000100:00080000:3.0:1787696472.723098:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696472.723100:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696472.723110:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696472.723120:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134251008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696472.723128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696472.723131:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696472.723132:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696472.723144:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4374 00000100:00080000:3.0:1787696472.723147:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696472.723150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696472.723153:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696472.723161:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134251136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696472.723168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696472.723171:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696472.723172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696472.723174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4374 00000100:00080000:3.0:1787696472.723176:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696472.723178:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696472.723182:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696472.723192:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134251264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696472.723199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696472.723201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696472.723202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696472.723204:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4374 00000100:00080000:3.0:1787696472.723207:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696472.723209:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696472.723212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696472.723220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134251392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696472.723226:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696472.723230:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696472.723233:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696472.723238:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4374 00000100:00080000:3.0:1787696472.723241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696472.723244:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696472.723247:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696472.723253:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134251520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696472.723260:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696472.723262:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696472.723264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696472.723267:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4374 00000100:00080000:3.0:1787696472.723269:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696472.723272:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696472.723275:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696472.723297:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134251648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696473.299126:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696474.323079:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696475.347008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696476.371037:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696477.843052:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696477.843077:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696477.843082:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696477.843092:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4379 00000100:00080000:3.0:1787696477.843113:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696477.843133:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696477.843207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134251776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696477.843259:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696477.843274:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696477.843277:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696477.843282:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4379 00000100:00080000:3.0:1787696477.843321:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696477.843328:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696477.843374:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696477.843467:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134251904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696477.843501:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696477.843505:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696477.843507:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696477.843521:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4379 00000100:00080000:3.0:1787696477.843526:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696477.843545:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696477.843600:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696477.843628:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134252032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696477.843644:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696477.843662:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696477.843664:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696477.843671:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4379 00000100:00080000:3.0:1787696477.843683:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696477.843688:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696477.843695:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696477.843714:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134252160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696477.843735:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696477.843744:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696477.843762:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696477.843770:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4379 00000100:00080000:3.0:1787696477.843777:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696477.843799:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696477.843815:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696477.843830:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134252288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696477.843887:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696477.843891:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696477.843892:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696477.843897:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4379 00000100:00080000:3.0:1787696477.843901:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696477.843923:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696477.843942:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696477.844015:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134252416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696477.844103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696477.844127:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696477.844131:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696477.844169:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4379 00000100:00080000:3.0:1787696477.844173:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696477.844177:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696477.844182:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696477.844216:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134252544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696478.419036:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696479.443007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696480.467048:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696481.491019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696482.514990:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696482.962919:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696482.962927:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696482.962930:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696482.962938:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4384 00000100:00080000:3.0:1787696482.962947:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696482.962953:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696482.962993:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134252672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696482.963024:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696482.963027:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696482.963028:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696482.963031:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4384 00000100:00080000:3.0:1787696482.963035:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696482.963038:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696482.963046:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696482.963058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134252800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696482.963068:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696482.963078:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696482.963079:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696482.963083:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4384 00000100:00080000:3.0:1787696482.963085:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696482.963088:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696482.963092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696482.963100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134252928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696482.963121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696482.963125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696482.963126:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696482.963130:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4384 00000100:00080000:3.0:1787696482.963133:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696482.963136:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696482.963140:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696482.963151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134253056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696482.963163:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696482.963166:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696482.963168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696482.963172:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4384 00000100:00080000:3.0:1787696482.963175:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696482.963178:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696482.963181:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696482.963187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134253184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696482.963198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696482.963201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696482.963202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696482.963204:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4384 00000100:00080000:3.0:1787696482.963207:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696482.963209:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696482.963212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696482.963221:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134253312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696482.963252:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696482.963255:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696482.963257:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696482.963262:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4384 00000100:00080000:3.0:1787696482.963265:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696482.963271:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696482.963278:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696482.963287:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134253440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696484.563010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696485.587052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696486.611000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696487.635049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696488.082939:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696488.082950:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696488.082954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696488.082966:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4389 00000100:00080000:3.0:1787696488.082976:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696488.082985:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696488.083053:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134253568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696488.083092:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696488.083097:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696488.083099:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696488.083104:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4389 00000100:00080000:3.0:1787696488.083108:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696488.083114:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696488.083125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696488.083142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134253696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696488.083154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696488.083159:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696488.083161:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696488.083165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4389 00000100:00080000:3.0:1787696488.083169:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696488.083195:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696488.083205:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696488.083224:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134253824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696488.083245:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696488.083249:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696488.083251:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696488.083256:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4389 00000100:00080000:3.0:1787696488.083261:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696488.083266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696488.083281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696488.083299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134253952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696488.083313:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696488.083318:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696488.083320:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696488.083324:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4389 00000100:00080000:3.0:1787696488.083329:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696488.083335:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696488.083341:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696488.083366:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134254080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696488.083432:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696488.083457:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696488.083462:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696488.083471:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4389 00000100:00080000:3.0:1787696488.083485:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696488.083498:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696488.083520:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696488.083556:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134254208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696488.083570:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696488.083574:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696488.083577:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696488.083597:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4389 00000100:00080000:3.0:1787696488.083614:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696488.083625:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696488.083643:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696488.083672:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134254336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696488.659057:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696490.707049:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696491.730975:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696492.755080:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696493.202989:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696493.203012:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696493.203016:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696493.203031:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4394 00000100:00080000:3.0:1787696493.203041:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696493.203047:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696493.203086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134254464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696493.203113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696493.203117:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696493.203119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696493.203124:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4394 00000100:00080000:3.0:1787696493.203127:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696493.203130:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696493.203148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696493.203162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134254592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696493.203175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696493.203178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696493.203180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696493.203191:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4394 00000100:00080000:3.0:1787696493.203194:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696493.203198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696493.203203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696493.203213:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134254720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696493.203221:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696493.203226:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696493.203231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696493.203235:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4394 00000100:00080000:3.0:1787696493.203239:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696493.203243:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696493.203248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696493.203258:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134254848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696493.203269:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696493.203276:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696493.203280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696493.203286:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4394 00000100:00080000:3.0:1787696493.203290:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696493.203297:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696493.203303:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696493.203316:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134254976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696493.203326:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696493.203329:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696493.203333:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696493.203341:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4394 00000100:00080000:3.0:1787696493.203344:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696493.203348:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696493.203352:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696493.203366:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134255104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696493.203393:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696493.203399:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696493.203403:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696493.203420:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4394 00000100:00080000:3.0:1787696493.203424:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696493.203428:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696493.203434:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696493.203448:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134255232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696493.779006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696494.803050:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696496.851028:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696497.875034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696498.322969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696498.322979:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696498.322983:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696498.322994:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4400 00000100:00080000:3.0:1787696498.323005:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696498.323013:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696498.323046:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134255360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696498.323075:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696498.323078:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696498.323080:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696498.323086:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4400 00000100:00080000:3.0:1787696498.323089:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696498.323093:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696498.323102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696498.323137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134255488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696498.323149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696498.323153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696498.323154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696498.323159:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4400 00000100:00080000:3.0:1787696498.323163:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696498.323166:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696498.323171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696498.323181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134255616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696498.323190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696498.323196:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696498.323200:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696498.323204:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4400 00000100:00080000:3.0:1787696498.323208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696498.323212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696498.323217:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696498.323229:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134255744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696498.323237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696498.323240:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696498.323242:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696498.323248:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4400 00000100:00080000:3.0:1787696498.323252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696498.323260:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696498.323269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696498.323282:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134255872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696498.323298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696498.323303:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696498.323308:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696498.323314:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4400 00000100:00080000:3.0:1787696498.323320:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696498.323323:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696498.323328:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696498.323340:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134256000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696498.323353:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696498.323356:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696498.323358:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696498.323361:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4400 00000100:00080000:3.0:1787696498.323376:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696498.323386:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696498.323395:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696498.323410:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134256128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696498.898985:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696499.922990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696500.947023:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696502.995008:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696503.442936:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696503.442945:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696503.442948:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696503.442957:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4405 00000100:00080000:3.0:1787696503.442966:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696503.442972:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696503.443005:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134256256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696503.443030:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696503.443033:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696503.443035:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696503.443038:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4405 00000100:00080000:3.0:1787696503.443041:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696503.443045:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696503.443053:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696503.443065:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134256384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696503.443075:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696503.443081:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696503.443084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696503.443088:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4405 00000100:00080000:3.0:1787696503.443091:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696503.443095:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696503.443100:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696503.443110:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134256512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696503.443119:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696503.443122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696503.443124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696503.443127:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4405 00000100:00080000:3.0:1787696503.443130:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696503.443136:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696503.443143:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696503.443155:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134256640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696503.443165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696503.443168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696503.443169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696503.443176:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4405 00000100:00080000:3.0:1787696503.443179:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696503.443187:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696503.443194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696503.443215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134256768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696503.443234:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696503.443239:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696503.443241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696503.443245:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4405 00000100:00080000:3.0:1787696503.443249:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696503.443256:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696503.443263:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696503.443275:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134256896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696503.443289:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696503.443295:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696503.443298:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696503.443307:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4405 00000100:00080000:3.0:1787696503.443312:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696503.443319:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696503.443333:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696503.443348:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134257024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696504.019025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696505.042990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696506.067098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696507.090987:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696508.562961:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696508.562990:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696508.562994:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696508.563007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4410 00000100:00080000:0.0:1787696508.563018:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696508.563024:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696508.563058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134257152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696508.563084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696508.563088:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696508.563089:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696508.563093:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4410 00000100:00080000:0.0:1787696508.563096:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696508.563099:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696508.563108:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696508.563120:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134257280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696508.563133:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696508.563137:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696508.563139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696508.563143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4410 00000100:00080000:0.0:1787696508.563146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696508.563150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696508.563155:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696508.563165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134257408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696508.563173:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696508.563177:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696508.563178:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696508.563182:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4410 00000100:00080000:0.0:1787696508.563186:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696508.563189:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696508.563194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696508.563206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134257536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696508.563215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696508.563218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696508.563219:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696508.563223:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4410 00000100:00080000:0.0:1787696508.563226:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696508.563230:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696508.563234:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696508.563248:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134257664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696508.563272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696508.563280:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696508.563284:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696508.563292:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4410 00000100:00080000:0.0:1787696508.563296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696508.563300:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696508.563305:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696508.563321:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134257792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696508.563331:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696508.563338:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696508.563340:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696508.563344:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4410 00000100:00080000:0.0:1787696508.563349:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696508.563357:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696508.563362:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696508.563379:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134257920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696509.139024:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696510.162989:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696511.186979:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696512.211020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696513.234976:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696513.682933:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696513.682940:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696513.682943:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696513.682951:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4415 00000100:00080000:3.0:1787696513.682960:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696513.682966:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696513.683001:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134258048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696513.683025:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696513.683028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696513.683029:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696513.683032:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4415 00000100:00080000:3.0:1787696513.683035:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696513.683038:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696513.683045:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696513.683055:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134258176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696513.683062:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696513.683064:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696513.683065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696513.683071:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4415 00000100:00080000:3.0:1787696513.683078:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696513.683082:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696513.683087:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696513.683099:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134258304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696513.683114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696513.683118:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696513.683119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696513.683123:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4415 00000100:00080000:3.0:1787696513.683128:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696513.683130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696513.683134:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696513.683145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134258432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696513.683154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696513.683160:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696513.683163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696513.683166:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4415 00000100:00080000:3.0:1787696513.683168:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696513.683170:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696513.683174:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696513.683188:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134258560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696513.683196:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696513.683199:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696513.683200:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696513.683205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4415 00000100:00080000:3.0:1787696513.683215:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696513.683219:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696513.683223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696513.683234:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134258688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696513.683242:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696513.683245:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696513.683245:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696513.683248:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4415 00000100:00080000:3.0:1787696513.683251:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696513.683255:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696513.683261:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696513.683275:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134258816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696515.283005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696516.306985:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787696516.755086:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787696516.755121:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787696516.755137:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696516.755184:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696516.755200:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696516.755209:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696516.755218:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696516.755227:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787696516.755235:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787696517.330984:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696518.355028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696518.802975:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696518.802987:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696518.802991:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696518.803002:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4420 00000100:00080000:3.0:1787696518.803015:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696518.803023:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696518.803064:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134258944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696518.803099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696518.803102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696518.803103:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696518.803107:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4420 00000100:00080000:3.0:1787696518.803110:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696518.803112:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696518.803122:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696518.803132:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134259072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696518.803139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696518.803142:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696518.803143:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696518.803146:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4420 00000100:00080000:3.0:1787696518.803148:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696518.803150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696518.803154:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696518.803162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134259200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696518.803174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696518.803176:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696518.803177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696518.803180:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4420 00000100:00080000:3.0:1787696518.803182:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696518.803185:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696518.803187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696518.803197:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134259328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696518.803203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696518.803205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696518.803206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696518.803208:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4420 00000100:00080000:3.0:1787696518.803211:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696518.803214:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696518.803219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696518.803226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134259456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696518.803232:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696518.803235:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696518.803237:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696518.803239:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4420 00000100:00080000:3.0:1787696518.803242:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696518.803244:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696518.803247:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696518.803256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134259584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696518.803274:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696518.803278:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696518.803280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696518.803284:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4420 00000100:00080000:3.0:1787696518.803288:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696518.803292:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696518.803297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696518.803311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134259712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696519.379096:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696521.427069:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696522.451036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696523.475006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696523.922928:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696523.922936:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696523.922939:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696523.922948:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4425 00000100:00080000:3.0:1787696523.922957:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696523.922975:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696523.923010:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134259840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696523.923033:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696523.923036:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696523.923037:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696523.923040:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4425 00000100:00080000:3.0:1787696523.923042:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696523.923045:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696523.923052:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696523.923059:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134259968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696523.923066:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696523.923068:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696523.923069:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696523.923071:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4425 00000100:00080000:3.0:1787696523.923073:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696523.923075:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696523.923078:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696523.923083:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134260096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696523.923088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696523.923090:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696523.923092:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696523.923094:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4425 00000100:00080000:3.0:1787696523.923096:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696523.923098:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696523.923101:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696523.923108:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134260224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696523.923114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696523.923119:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696523.923123:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696523.923130:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4425 00000100:00080000:3.0:1787696523.923132:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696523.923134:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696523.923137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696523.923145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134260352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696523.923151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696523.923154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696523.923155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696523.923158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4425 00000100:00080000:3.0:1787696523.923161:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696523.923163:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696523.923166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696523.923171:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134260480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696523.923177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696523.923179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696523.923181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696523.923188:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4425 00000100:00080000:3.0:1787696523.923192:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696523.923194:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696523.923198:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696523.923207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134260608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696524.499012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696525.522980:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696527.571035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696528.595029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696529.042957:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696529.042968:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696529.042972:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696529.042984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4430 00000100:00080000:3.0:1787696529.042997:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696529.043006:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696529.043052:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134260736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696529.043081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696529.043084:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696529.043085:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696529.043089:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4430 00000100:00080000:3.0:1787696529.043092:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696529.043094:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696529.043103:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696529.043114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134260864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696529.043122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696529.043125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696529.043127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696529.043132:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4430 00000100:00080000:3.0:1787696529.043135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696529.043141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696529.043147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696529.043153:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134260992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696529.043168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696529.043172:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696529.043174:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696529.043179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4430 00000100:00080000:3.0:1787696529.043184:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696529.043189:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696529.043194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696529.043211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134261120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696529.043220:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696529.043222:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696529.043224:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696529.043229:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4430 00000100:00080000:3.0:1787696529.043236:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696529.043241:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696529.043245:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696529.043256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134261248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696529.043263:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696529.043265:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696529.043266:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696529.043269:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4430 00000100:00080000:3.0:1787696529.043271:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696529.043273:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696529.043277:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696529.043284:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134261376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696529.043290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696529.043292:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696529.043294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696529.043297:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4430 00000100:00080000:3.0:1787696529.043299:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696529.043302:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696529.043305:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696529.043332:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134261504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696529.619102:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696530.643016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696531.666991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696533.715050:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696534.162919:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696534.162926:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696534.162930:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696534.162938:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4435 00000100:00080000:0.0:1787696534.162947:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696534.162953:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696534.162984:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134261632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696534.163004:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696534.163007:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696534.163008:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696534.163011:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4435 00000100:00080000:0.0:1787696534.163014:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696534.163016:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696534.163025:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696534.163035:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134261760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696534.163046:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696534.163051:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696534.163052:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696534.163056:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4435 00000100:00080000:0.0:1787696534.163058:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696534.163061:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696534.163065:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696534.163081:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134261888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696534.163088:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696534.163091:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696534.163092:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696534.163094:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4435 00000100:00080000:0.0:1787696534.163102:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696534.163105:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696534.163112:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696534.163123:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134262016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696534.163130:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696534.163133:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696534.163134:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696534.163137:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4435 00000100:00080000:0.0:1787696534.163141:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696534.163144:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696534.163148:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696534.163161:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134262144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696534.163168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696534.163175:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696534.163178:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696534.163187:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4435 00000100:00080000:0.0:1787696534.163190:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696534.163197:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696534.163206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696534.163234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134262272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696534.163253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696534.163257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696534.163260:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696534.163267:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4435 00000100:00080000:0.0:1787696534.163276:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696534.163285:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696534.163295:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696534.163309:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134262400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696534.739032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696535.763030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696536.787044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696537.811086:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696539.282965:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696539.282977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696539.282981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696539.283001:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4441 00000100:00080000:0.0:1787696539.283013:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696539.283020:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696539.283058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134262528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696539.283092:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696539.283096:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696539.283098:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696539.283103:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4441 00000100:00080000:0.0:1787696539.283107:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696539.283110:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696539.283121:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696539.283149:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134262656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696539.283165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696539.283169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696539.283171:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696539.283177:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4441 00000100:00080000:0.0:1787696539.283182:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696539.283187:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696539.283194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696539.283207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134262784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696539.283219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696539.283224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696539.283226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696539.283233:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4441 00000100:00080000:0.0:1787696539.283237:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696539.283241:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696539.283246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696539.283262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134262912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696539.283277:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696539.283286:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696539.283290:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696539.283299:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4441 00000100:00080000:0.0:1787696539.283305:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696539.283315:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696539.283321:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696539.283333:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134263040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696539.283352:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696539.283357:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696539.283362:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696539.283369:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4441 00000100:00080000:0.0:1787696539.283374:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696539.283382:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696539.283388:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696539.283410:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134263168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696539.283448:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696539.283451:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696539.283453:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696539.283458:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4441 00000100:00080000:0.0:1787696539.283464:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696539.283473:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696539.283481:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696539.283497:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134263296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696539.859068:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696540.882991:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696541.907010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696542.930990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696543.955039:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696544.402985:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696544.402995:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696544.403000:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696544.403010:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4446 00000100:00080000:3.0:1787696544.403021:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696544.403029:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696544.403101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134263424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696544.403131:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696544.403135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696544.403137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696544.403141:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4446 00000100:00080000:3.0:1787696544.403146:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696544.403151:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696544.403160:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696544.403178:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134263552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696544.403208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696544.403212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696544.403214:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696544.403218:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4446 00000100:00080000:3.0:1787696544.403223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696544.403228:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696544.403240:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696544.403256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134263680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696544.403267:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696544.403273:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696544.403278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696544.403299:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4446 00000100:00080000:3.0:1787696544.403305:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696544.403316:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696544.403326:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696544.403343:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134263808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696544.403357:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696544.403365:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696544.403370:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696544.403378:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4446 00000100:00080000:3.0:1787696544.403386:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696544.403394:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696544.403405:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696544.403425:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134263936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696544.403437:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696544.403444:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696544.403447:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696544.403456:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4446 00000100:00080000:3.0:1787696544.403464:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696544.403479:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696544.403489:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696544.403510:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134264064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696544.403525:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696544.403530:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696544.403535:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696544.403547:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4446 00000100:00080000:3.0:1787696544.403554:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696544.403564:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696544.403573:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696544.403593:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134264192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787696544.978973:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696546.003052:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696547.027048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696548.051073:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696549.075045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696549.522980:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696549.522989:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696549.522992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696549.523000:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4451 00000100:00080000:3.0:1787696549.523010:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696549.523016:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696549.523050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134264320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696549.523075:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696549.523078:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696549.523079:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696549.523082:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4451 00000100:00080000:3.0:1787696549.523084:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696549.523087:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696549.523093:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696549.523102:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134264448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696549.523109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696549.523112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696549.523113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696549.523116:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4451 00000100:00080000:3.0:1787696549.523118:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696549.523120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696549.523124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696549.523134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134264576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696549.523142:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696549.523144:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696549.523145:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696549.523149:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4451 00000100:00080000:3.0:1787696549.523151:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696549.523153:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696549.523156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696549.523166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134264704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696549.523171:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696549.523174:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696549.523175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696549.523178:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4451 00000100:00080000:3.0:1787696549.523180:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696549.523183:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696549.523186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696549.523196:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134264832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696549.523204:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696549.523207:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696549.523208:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696549.523210:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4451 00000100:00080000:3.0:1787696549.523213:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696549.523215:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696549.523219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696549.523228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134264960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696549.523252:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696549.523256:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696549.523260:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696549.523270:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4451 00000100:00080000:3.0:1787696549.523278:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696549.523285:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696549.523296:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696549.523309:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134265088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696550.099028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696551.123020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696552.146988:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696553.171019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696554.195006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696554.642968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696554.642979:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696554.642984:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696554.642992:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4456 00000100:00080000:0.0:1787696554.643002:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696554.643008:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696554.643040:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134265216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696554.643063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696554.643066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696554.643067:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696554.643074:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4456 00000100:00080000:0.0:1787696554.643078:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696554.643081:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696554.643090:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696554.643099:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134265344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696554.643107:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696554.643109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696554.643110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696554.643113:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4456 00000100:00080000:0.0:1787696554.643116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696554.643123:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696554.643131:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696554.643145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134265472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696554.643157:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696554.643161:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696554.643163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696554.643166:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4456 00000100:00080000:0.0:1787696554.643172:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696554.643175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696554.643183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696554.643196:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134265600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696554.643208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696554.643211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696554.643212:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696554.643218:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4456 00000100:00080000:0.0:1787696554.643221:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696554.643228:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696554.643236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696554.643247:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134265728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696554.643267:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696554.643273:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696554.643276:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696554.643282:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4456 00000100:00080000:0.0:1787696554.643289:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696554.643299:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696554.643307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696554.643323:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134265856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696554.643349:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696554.643354:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696554.643358:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696554.643367:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4456 00000100:00080000:0.0:1787696554.643372:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696554.643380:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696554.643386:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696554.643398:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134265984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696555.219004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696556.242989:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696557.266993:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696558.291003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696559.315014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696559.762978:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696559.762986:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696559.762989:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696559.762998:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4461 00000100:00080000:3.0:1787696559.763007:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696559.763013:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696559.763055:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134266112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696559.763077:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696559.763080:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696559.763081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696559.763086:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4461 00000100:00080000:3.0:1787696559.763088:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696559.763091:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696559.763098:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696559.763106:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134266240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696559.763112:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696559.763115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696559.763116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696559.763118:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4461 00000100:00080000:3.0:1787696559.763120:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696559.763122:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696559.763124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696559.763129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134266368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696559.763136:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696559.763138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696559.763140:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696559.763142:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4461 00000100:00080000:3.0:1787696559.763145:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696559.763149:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696559.763152:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696559.763158:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134266496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696559.763168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696559.763173:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696559.763175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696559.763177:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4461 00000100:00080000:3.0:1787696559.763179:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696559.763181:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696559.763185:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696559.763193:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134266624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696559.763205:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696559.763208:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696559.763209:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696559.763211:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4461 00000100:00080000:3.0:1787696559.763214:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696559.763216:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696559.763219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696559.763228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134266752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696559.763240:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696559.763242:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696559.763244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696559.763248:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4461 00000100:00080000:3.0:1787696559.763251:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696559.763254:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696559.763260:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696559.763276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134266880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696560.339031:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696561.363045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696562.387024:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696563.411015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696564.435005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696564.882969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696564.882977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696564.882980:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696564.882988:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4466 00000100:00080000:0.0:1787696564.882997:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696564.883003:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696564.883036:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134267008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696564.883058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696564.883061:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696564.883063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696564.883070:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4466 00000100:00080000:0.0:1787696564.883072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696564.883075:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696564.883084:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696564.883098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134267136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696564.883120:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696564.883126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696564.883128:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696564.883134:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4466 00000100:00080000:0.0:1787696564.883138:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696564.883143:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696564.883149:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696564.883163:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134267264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696564.883174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696564.883178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696564.883181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696564.883184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4466 00000100:00080000:0.0:1787696564.883189:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696564.883196:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696564.883204:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696564.883219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134267392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696564.883231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696564.883236:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696564.883238:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696564.883246:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4466 00000100:00080000:0.0:1787696564.883255:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696564.883262:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696564.883268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696564.883284:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134267520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696564.883298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696564.883305:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696564.883310:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696564.883318:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4466 00000100:00080000:0.0:1787696564.883324:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696564.883328:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696564.883333:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696564.883344:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134267648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696564.883351:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696564.883366:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696564.883368:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696564.883370:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4466 00000100:00080000:0.0:1787696564.883380:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696564.883385:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696564.883390:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696564.883402:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134267776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696565.459046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696566.482984:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696567.507000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696568.531047:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696569.555004:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696570.002960:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696570.002971:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696570.002976:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696570.002987:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4471 00000100:00080000:3.0:1787696570.003001:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696570.003009:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696570.003053:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134267904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696570.003084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696570.003088:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696570.003090:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696570.003096:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4471 00000100:00080000:3.0:1787696570.003100:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696570.003103:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696570.003115:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696570.003137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134268032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696570.003155:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696570.003160:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696570.003163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696570.003169:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4471 00000100:00080000:3.0:1787696570.003175:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696570.003181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696570.003191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696570.003215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134268160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696570.003238:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696570.003244:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696570.003246:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696570.003254:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4471 00000100:00080000:3.0:1787696570.003262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696570.003278:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696570.003285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696570.003306:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134268288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696570.003337:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696570.003341:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696570.003344:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696570.003347:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4471 00000100:00080000:3.0:1787696570.003353:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696570.003356:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696570.003361:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696570.003372:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134268416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696570.003389:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696570.003392:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696570.003393:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696570.003397:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4471 00000100:00080000:3.0:1787696570.003420:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696570.003434:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696570.003448:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696570.003476:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134268544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696570.003505:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696570.003509:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696570.003512:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696570.003519:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4471 00000100:00080000:3.0:1787696570.003524:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696570.003531:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696570.003538:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696570.003558:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134268672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696570.579031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696571.602991:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696572.627032:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696573.651003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696574.675019:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696575.122927:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696575.122936:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696575.122941:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696575.122951:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4476 00000100:00080000:3.0:1787696575.122960:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696575.122966:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696575.122996:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134268800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696575.123017:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696575.123020:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696575.123021:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696575.123024:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4476 00000100:00080000:3.0:1787696575.123026:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696575.123029:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696575.123036:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696575.123044:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134268928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696575.123053:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696575.123057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696575.123060:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696575.123065:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4476 00000100:00080000:3.0:1787696575.123071:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696575.123074:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696575.123077:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696575.123084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134269056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696575.123091:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696575.123093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696575.123095:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696575.123097:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4476 00000100:00080000:3.0:1787696575.123099:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696575.123102:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696575.123105:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696575.123112:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134269184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696575.123117:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696575.123120:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696575.123122:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696575.123128:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4476 00000100:00080000:3.0:1787696575.123133:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696575.123136:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696575.123140:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696575.123147:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134269312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696575.123158:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696575.123162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696575.123164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696575.123172:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4476 00000100:00080000:3.0:1787696575.123177:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696575.123183:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696575.123186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696575.123195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134269440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696575.123215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696575.123222:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696575.123231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696575.123238:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4476 00000100:00080000:3.0:1787696575.123244:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696575.123249:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696575.123254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696575.123263:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134269568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696575.699045:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696576.723039:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696577.747018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696578.771006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696579.794977:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696580.242939:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696580.242949:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696580.242954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696580.242964:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4482 00000100:00080000:3.0:1787696580.242975:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696580.242982:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696580.243022:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134269696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696580.243053:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696580.243057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696580.243059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696580.243063:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4482 00000100:00080000:3.0:1787696580.243067:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696580.243071:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696580.243081:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696580.243095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134269824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696580.243105:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696580.243109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696580.243112:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696580.243116:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4482 00000100:00080000:3.0:1787696580.243120:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696580.243124:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696580.243129:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696580.243141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134269952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696580.243160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696580.243163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696580.243165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696580.243168:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4482 00000100:00080000:3.0:1787696580.243171:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696580.243175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696580.243180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696580.243192:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134270080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696580.243201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696580.243205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696580.243206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696580.243209:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4482 00000100:00080000:3.0:1787696580.243213:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696580.243218:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696580.243223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696580.243236:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134270208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696580.243253:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696580.243257:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696580.243260:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696580.243266:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4482 00000100:00080000:3.0:1787696580.243272:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696580.243277:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696580.243283:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696580.243302:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134270336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696580.243324:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696580.243328:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696580.243330:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696580.243337:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4482 00000100:00080000:3.0:1787696580.243343:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696580.243349:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696580.243355:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696580.243376:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134270464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696580.819027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696581.843047:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696582.867078:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696583.315227:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696583.315281:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696583.315304:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696583.315353:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696583.315365:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696583.315414:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696583.315428:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696583.315447:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696583.315472:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:3.0:1787696583.891012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696584.914996:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696585.362928:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696585.362936:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696585.362939:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696585.362948:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4487 00000100:00080000:3.0:1787696585.362958:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696585.362963:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696585.362993:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134270592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696585.363017:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696585.363022:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696585.363024:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696585.363028:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4487 00000100:00080000:3.0:1787696585.363032:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696585.363036:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696585.363046:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696585.363060:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134270720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696585.363072:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696585.363076:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696585.363078:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696585.363083:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4487 00000100:00080000:3.0:1787696585.363087:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696585.363092:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696585.363098:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696585.363124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134270848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696585.363142:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696585.363146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696585.363149:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696585.363153:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4487 00000100:00080000:3.0:1787696585.363157:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696585.363160:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696585.363164:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696585.363175:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134270976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696585.363187:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696585.363190:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696585.363191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696585.363194:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4487 00000100:00080000:3.0:1787696585.363198:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696585.363200:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696585.363206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696585.363217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134271104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696585.363227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696585.363231:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696585.363233:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696585.363238:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4487 00000100:00080000:3.0:1787696585.363244:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696585.363249:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696585.363255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696585.363272:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134271232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696585.363284:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696585.363290:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696585.363293:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696585.363302:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4487 00000100:00080000:3.0:1787696585.363308:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696585.363313:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696585.363318:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696585.363331:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134271360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696585.939036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696586.963051:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696587.987081:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696589.011100:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696590.035010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696590.483001:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696590.483011:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696590.483015:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696590.483024:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4492 00000100:00080000:2.0:1787696590.483033:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696590.483038:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696590.483070:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134271488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696590.483093:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696590.483096:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696590.483098:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696590.483102:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4492 00000100:00080000:2.0:1787696590.483106:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696590.483118:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696590.483127:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696590.483140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134271616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696590.483151:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696590.483154:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696590.483156:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696590.483159:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4492 00000100:00080000:2.0:1787696590.483162:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696590.483165:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696590.483170:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696590.483178:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134271744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696590.483187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696590.483190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696590.483192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696590.483195:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4492 00000100:00080000:2.0:1787696590.483198:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696590.483202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696590.483206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696590.483215:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134271872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696590.483224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696590.483227:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696590.483228:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696590.483232:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4492 00000100:00080000:2.0:1787696590.483235:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696590.483239:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696590.483244:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696590.483282:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134272000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696590.483294:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696590.483298:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696590.483300:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696590.483304:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4492 00000100:00080000:2.0:1787696590.483308:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696590.483311:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696590.483316:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696590.483330:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134272128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696590.483340:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696590.483343:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696590.483344:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696590.483348:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4492 00000100:00080000:2.0:1787696590.483351:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696590.483355:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696590.483359:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696590.483369:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134272256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696591.059034:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696592.083111:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696593.106984:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696594.131095:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696595.155020:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696595.603217:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134272384:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696595.603324:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696595.603387:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696595/real 1787696595] req@ffff9d85b9d2c700 x1874531134272384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696595.603467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134272384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696595.603481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696595.603486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696595.603490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134272384:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696595.603498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134272512:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696595.603523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696595.603535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696595/real 1787696595] req@ffff9d85b9d2ed80 x1874531134272512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696595.603553:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134272512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696595.603561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696595.603564:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696595.603568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134272512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696595.603603:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134272640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696595.603625:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696595.603642:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696595/real 1787696595] req@ffff9d85b9d2d500 x1874531134272640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696595.603653:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134272640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696595.603661:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696595.603665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696595.603668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134272640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696595.603674:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134272768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696595.603693:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696595.603704:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696595/real 1787696595] req@ffff9d85b9d2f800 x1874531134272768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696595.603714:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134272768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696595.603727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696595.603730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696595.603733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134272768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696595.603739:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134272896:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696595.603756:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696595.603766:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696595/real 1787696595] req@ffff9d85b9d2dc00 x1874531134272896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696595.603777:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134272896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696595.603784:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696595.603788:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696595.603791:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134272896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696595.603796:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134273024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696595.603812:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696595.603821:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696595/real 1787696595] req@ffff9d85b9d2e680 x1874531134273024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696595.603832:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134273024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696595.603839:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696595.603842:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696595.603846:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134273024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696595.603853:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134273152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696595.603912:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696595.603924:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696595/real 1787696595] req@ffff9d85b9d2c380 x1874531134273152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696595.603934:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134273152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696595.603943:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696595.603946:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696595.603949:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134273152:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696596.179028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696597.203034:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696598.227043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696599.251019:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696600.275051:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696600.722943:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696600.722962:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696600.722967:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696600.722976:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4502 00000100:00080000:3.0:1787696600.722987:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696600.722994:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696600.723030:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134273280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696600.723066:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696600.723074:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696600.723089:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696600.723095:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4502 00000100:00080000:3.0:1787696600.723099:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696600.723103:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696600.723114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696600.723132:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134273408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696600.723142:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696600.723145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696600.723147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696600.723151:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4502 00000100:00080000:3.0:1787696600.723157:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696600.723161:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696600.723167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696600.723193:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134273536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696600.723223:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696600.723227:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696600.723231:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696600.723240:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4502 00000100:00080000:3.0:1787696600.723245:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696600.723254:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696600.723262:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696600.723284:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134273664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696600.723307:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696600.723312:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696600.723314:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696600.723328:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4502 00000100:00080000:3.0:1787696600.723331:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696600.723335:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696600.723339:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696600.723352:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134273792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696600.723362:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696600.723365:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696600.723367:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696600.723370:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4502 00000100:00080000:3.0:1787696600.723373:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696600.723376:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696600.723380:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696600.723390:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134273920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696600.723398:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696600.723401:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696600.723405:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696600.723412:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4502 00000100:00080000:3.0:1787696600.723418:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696600.723424:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696600.723429:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696600.723437:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134274048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696601.298997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696602.323085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696603.346987:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696604.371024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696605.394981:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696605.842958:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696605.842966:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696605.842969:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696605.842978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4507 00000100:00080000:0.0:1787696605.843007:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696605.843013:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696605.843047:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134274176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696605.843070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696605.843073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696605.843075:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696605.843077:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4507 00000100:00080000:0.0:1787696605.843080:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696605.843082:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696605.843090:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696605.843099:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134274304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696605.843107:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696605.843110:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696605.843111:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696605.843114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4507 00000100:00080000:0.0:1787696605.843117:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696605.843122:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696605.843130:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696605.843138:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134274432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696605.843145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696605.843148:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696605.843149:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696605.843152:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4507 00000100:00080000:0.0:1787696605.843155:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696605.843157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696605.843160:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696605.843167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134274560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696605.843173:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696605.843175:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696605.843177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696605.843182:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4507 00000100:00080000:0.0:1787696605.843187:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696605.843191:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696605.843194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696605.843203:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134274688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696605.843209:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696605.843212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696605.843214:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696605.843217:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4507 00000100:00080000:0.0:1787696605.843221:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696605.843227:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696605.843233:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696605.843246:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134274816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696605.843263:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696605.843272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696605.843275:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696605.843282:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4507 00000100:00080000:0.0:1787696605.843285:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696605.843291:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696605.843296:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696605.843311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134274944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696606.419007:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696607.443102:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696608.466997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696609.491085:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696610.514988:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696610.962949:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696610.962958:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696610.962962:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696610.962971:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4512 00000100:00080000:3.0:1787696610.962981:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696610.962987:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696610.963021:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134275072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696610.963044:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696610.963047:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696610.963048:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696610.963051:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4512 00000100:00080000:3.0:1787696610.963053:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696610.963056:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696610.963064:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696610.963073:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134275200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696610.963081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696610.963083:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696610.963084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696610.963090:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4512 00000100:00080000:3.0:1787696610.963095:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696610.963101:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696610.963105:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696610.963114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134275328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696610.963121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696610.963124:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696610.963125:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696610.963127:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4512 00000100:00080000:3.0:1787696610.963131:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696610.963134:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696610.963137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696610.963144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134275456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696610.963150:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696610.963154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696610.963157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696610.963162:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4512 00000100:00080000:3.0:1787696610.963165:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696610.963169:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696610.963172:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696610.963181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134275584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696610.963187:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696610.963189:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696610.963190:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696610.963193:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4512 00000100:00080000:3.0:1787696610.963196:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696610.963198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696610.963204:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696610.963218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134275712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696610.963227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696610.963236:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696610.963239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696610.963247:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4512 00000100:00080000:3.0:1787696610.963252:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696610.963259:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696610.963262:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696610.963273:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134275840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696611.539044:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696612.563015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696613.587021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696614.611019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696615.635003:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696616.082982:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696616.082992:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696616.082997:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696616.083008:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4517 00000100:00080000:0.0:1787696616.083019:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696616.083027:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696616.083065:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134275968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696616.083089:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696616.083091:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696616.083093:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696616.083096:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4517 00000100:00080000:0.0:1787696616.083099:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696616.083101:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696616.083109:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696616.083118:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134276096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696616.083127:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696616.083129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696616.083131:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696616.083137:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4517 00000100:00080000:0.0:1787696616.083143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696616.083146:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696616.083150:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696616.083158:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134276224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696616.083165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696616.083167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696616.083169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696616.083172:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4517 00000100:00080000:0.0:1787696616.083190:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696616.083193:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696616.083196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696616.083206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134276352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696616.083214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696616.083216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696616.083217:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696616.083220:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4517 00000100:00080000:0.0:1787696616.083226:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696616.083228:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696616.083231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696616.083240:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134276480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696616.083246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696616.083251:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696616.083253:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696616.083260:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4517 00000100:00080000:0.0:1787696616.083266:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696616.083271:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696616.083275:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696616.083285:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134276608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696616.083322:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696616.083327:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696616.083332:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696616.083339:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4517 00000100:00080000:0.0:1787696616.083344:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696616.083347:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696616.083350:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696616.083357:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134276736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696616.658982:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696617.683008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696618.707002:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696619.731026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696620.755079:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696621.202950:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696621.202961:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696621.202966:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696621.202988:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4522 00000100:00080000:3.0:1787696621.202999:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696621.203007:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696621.203045:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134276864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696621.203068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696621.203070:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696621.203072:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696621.203075:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4522 00000100:00080000:3.0:1787696621.203079:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696621.203081:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696621.203089:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696621.203100:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134276992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696621.203110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696621.203118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696621.203122:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696621.203128:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4522 00000100:00080000:3.0:1787696621.203133:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696621.203137:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696621.203144:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696621.203159:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134277120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696621.203170:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696621.203179:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696621.203182:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696621.203187:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4522 00000100:00080000:3.0:1787696621.203191:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696621.203194:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696621.203198:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696621.203209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134277248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696621.203216:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696621.203219:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696621.203220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696621.203223:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4522 00000100:00080000:3.0:1787696621.203225:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696621.203227:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696621.203233:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696621.203246:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134277376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696621.203256:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696621.203259:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696621.203260:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696621.203266:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4522 00000100:00080000:3.0:1787696621.203271:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696621.203277:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696621.203283:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696621.203290:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134277504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696621.203296:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696621.203299:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696621.203301:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696621.203310:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4522 00000100:00080000:3.0:1787696621.203315:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696621.203321:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696621.203329:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696621.203336:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134277632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696621.779018:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696622.803017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696623.827027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696624.850970:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696625.875002:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696626.322952:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696626.322960:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696626.322963:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696626.322971:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4528 00000100:00080000:2.0:1787696626.322981:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696626.322987:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696626.323021:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134277760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696626.323059:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696626.323063:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696626.323065:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696626.323069:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4528 00000100:00080000:2.0:1787696626.323072:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696626.323074:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696626.323082:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696626.323097:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134277888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696626.323106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696626.323108:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696626.323110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696626.323114:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4528 00000100:00080000:2.0:1787696626.323118:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696626.323122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696626.323129:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696626.323137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134278016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696626.323143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696626.323146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696626.323147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696626.323149:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4528 00000100:00080000:2.0:1787696626.323150:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696626.323152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696626.323155:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696626.323163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134278144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696626.323168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696626.323171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696626.323172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696626.323175:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4528 00000100:00080000:2.0:1787696626.323177:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696626.323180:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696626.323183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696626.323189:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134278272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696626.323195:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696626.323198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696626.323199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696626.323201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4528 00000100:00080000:2.0:1787696626.323211:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696626.323216:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696626.323220:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696626.323228:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134278400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696626.323234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696626.323236:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696626.323237:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696626.323240:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4528 00000100:00080000:2.0:1787696626.323242:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696626.323244:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696626.323247:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696626.323254:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134278528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696626.899019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696627.923017:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696628.946998:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696629.970978:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696630.995013:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696631.442971:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696631.442980:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696631.442984:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696631.442992:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4533 00000100:00080000:3.0:1787696631.443001:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696631.443007:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696631.443040:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134278656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696631.443062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696631.443064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696631.443066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696631.443069:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4533 00000100:00080000:3.0:1787696631.443071:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696631.443073:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696631.443080:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696631.443087:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134278784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696631.443094:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696631.443096:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696631.443097:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696631.443099:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4533 00000100:00080000:3.0:1787696631.443100:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696631.443102:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696631.443105:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696631.443110:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134278912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696631.443121:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696631.443124:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696631.443125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696631.443128:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4533 00000100:00080000:3.0:1787696631.443130:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696631.443132:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696631.443135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696631.443143:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134279040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696631.443153:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696631.443156:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696631.443157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696631.443159:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4533 00000100:00080000:3.0:1787696631.443162:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696631.443164:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696631.443167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696631.443173:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134279168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696631.443179:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696631.443181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696631.443183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696631.443185:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4533 00000100:00080000:3.0:1787696631.443188:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696631.443191:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696631.443194:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696631.443201:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134279296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696631.443210:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696631.443213:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696631.443214:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696631.443218:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4533 00000100:00080000:3.0:1787696631.443221:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696631.443223:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696631.443226:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696631.443232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134279424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696632.018990:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696633.043028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696634.066962:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696635.090982:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696636.114975:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696636.562970:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696636.562978:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696636.562981:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696636.562990:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4538 00000100:00080000:2.0:1787696636.563000:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696636.563005:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696636.563040:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134279552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696636.563063:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696636.563067:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696636.563069:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696636.563073:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4538 00000100:00080000:2.0:1787696636.563078:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696636.563081:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696636.563091:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696636.563104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134279680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696636.563115:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696636.563118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696636.563120:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696636.563123:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4538 00000100:00080000:2.0:1787696636.563126:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696636.563130:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696636.563135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696636.563143:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134279808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696636.563152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696636.563155:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696636.563156:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696636.563159:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4538 00000100:00080000:2.0:1787696636.563162:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696636.563165:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696636.563169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696636.563179:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134279936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696636.563188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696636.563191:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696636.563192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696636.563195:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4538 00000100:00080000:2.0:1787696636.563198:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696636.563202:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696636.563206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696636.563215:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134280064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696636.563224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696636.563228:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696636.563229:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696636.563233:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4538 00000100:00080000:2.0:1787696636.563236:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696636.563240:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696636.563245:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696636.563253:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134280192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696636.563262:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696636.563266:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696636.563267:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696636.563271:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4538 00000100:00080000:2.0:1787696636.563274:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696636.563278:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696636.563282:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696636.563291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134280320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696637.139091:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696638.162995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696639.187121:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696640.210990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696641.235025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696641.682998:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696641.683008:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696641.683011:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696641.683019:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4543 00000100:00080000:3.0:1787696641.683029:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696641.683035:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696641.683068:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134280448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696641.683095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696641.683098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696641.683100:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696641.683103:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4543 00000100:00080000:3.0:1787696641.683105:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696641.683107:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696641.683115:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696641.683129:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134280576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696641.683136:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696641.683139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696641.683140:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696641.683142:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4543 00000100:00080000:3.0:1787696641.683144:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696641.683147:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696641.683152:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696641.683160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134280704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696641.683174:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696641.683176:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696641.683177:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696641.683179:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4543 00000100:00080000:3.0:1787696641.683182:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696641.683185:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696641.683188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696641.683198:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134280832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696641.683204:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696641.683206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696641.683207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696641.683209:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4543 00000100:00080000:3.0:1787696641.683212:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696641.683215:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696641.683218:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696641.683227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134280960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696641.683234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696641.683237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696641.683238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696641.683242:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4543 00000100:00080000:3.0:1787696641.683244:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696641.683246:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696641.683249:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696641.683256:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134281088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696641.683266:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696641.683268:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696641.683271:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696641.683280:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4543 00000100:00080000:3.0:1787696641.683285:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696641.683289:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696641.683293:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696641.683303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134281216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696642.258994:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696643.283011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696644.306987:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696645.331021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696646.354961:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696646.803060:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696646.803073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696646.803079:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696646.803093:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4548 00000100:00080000:0.0:1787696646.803108:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696646.803117:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696646.803184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134281344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696646.803217:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696646.803227:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696646.803230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696646.803235:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4548 00000100:00080000:0.0:1787696646.803239:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696646.803244:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696646.803257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696646.803272:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134281472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696646.803301:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696646.803304:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696646.803308:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696646.803314:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4548 00000100:00080000:0.0:1787696646.803318:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696646.803328:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696646.803342:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696646.803360:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134281600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696646.803372:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696646.803376:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696646.803378:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696646.803383:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4548 00000100:00080000:0.0:1787696646.803396:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696646.803401:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696646.803413:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696646.803428:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134281728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696646.803440:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696646.803458:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696646.803463:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696646.803471:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4548 00000100:00080000:0.0:1787696646.803483:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696646.803488:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696646.803495:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696646.803523:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134281856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696646.803558:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696646.803564:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696646.803567:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696646.803576:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4548 00000100:00080000:0.0:1787696646.803582:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696646.803589:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696646.803596:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696646.803619:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134281984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696646.803632:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696646.803637:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696646.803639:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696646.803646:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4548 00000100:00080000:0.0:1787696646.803660:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696646.803677:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696646.803685:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696646.803701:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134282112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696647.379057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696648.403142:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696649.427006:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696649.875091:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696649.875148:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696649.875163:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696649.875204:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696649.875214:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696649.875222:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696649.875245:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696649.875258:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696649.875271:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00010000:00010000:2.0:1787696650.451012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696651.474996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696651.922944:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696651.922956:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696651.922960:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696651.922970:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4553 00000100:00080000:3.0:1787696651.922982:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696651.922989:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696651.923027:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134282240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696651.923057:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696651.923060:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696651.923063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696651.923069:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4553 00000100:00080000:3.0:1787696651.923073:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696651.923077:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696651.923087:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696651.923101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134282368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696651.923114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696651.923119:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696651.923124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696651.923130:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4553 00000100:00080000:3.0:1787696651.923133:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696651.923137:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696651.923143:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696651.923157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134282496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696651.923167:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696651.923170:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696651.923172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696651.923177:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4553 00000100:00080000:3.0:1787696651.923181:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696651.923188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696651.923235:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696651.923250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134282624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696651.923263:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696651.923271:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696651.923275:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696651.923285:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4553 00000100:00080000:3.0:1787696651.923290:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696651.923298:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696651.923305:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696651.923324:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134282752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696651.923345:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696651.923350:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696651.923353:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696651.923358:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4553 00000100:00080000:3.0:1787696651.923362:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696651.923367:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696651.923374:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696651.923390:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134282880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696651.923403:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696651.923406:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696651.923411:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696651.923421:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4553 00000100:00080000:3.0:1787696651.923427:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696651.923432:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696651.923449:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696651.923465:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134283008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696652.499029:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696653.523071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696654.547042:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696655.571028:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696656.595015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696657.042974:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696657.042985:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696657.042990:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696657.043001:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4558 00000100:00080000:3.0:1787696657.043013:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696657.043020:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696657.043057:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134283136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696657.043088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696657.043093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696657.043095:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696657.043100:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4558 00000100:00080000:3.0:1787696657.043117:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696657.043124:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696657.043135:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696657.043152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134283264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696657.043168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696657.043172:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696657.043174:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696657.043179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4558 00000100:00080000:3.0:1787696657.043183:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696657.043188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696657.043195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696657.043209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134283392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696657.043224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696657.043229:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696657.043231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696657.043236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4558 00000100:00080000:3.0:1787696657.043241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696657.043245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696657.043251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696657.043265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134283520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696657.043275:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696657.043281:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696657.043285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696657.043294:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4558 00000100:00080000:3.0:1787696657.043301:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696657.043306:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696657.043314:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696657.043328:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134283648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696657.043351:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696657.043357:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696657.043359:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696657.043373:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4558 00000100:00080000:3.0:1787696657.043380:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696657.043385:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696657.043391:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696657.043405:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134283776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696657.043422:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696657.043429:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696657.043434:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696657.043442:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4558 00000100:00080000:3.0:1787696657.043448:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696657.043457:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696657.043465:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696657.043492:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134283904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696657.619030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696658.643015:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696659.667024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696660.691018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696661.714987:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696662.163104:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134284032:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696662.163184:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696662.163245:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696662/real 1787696662] req@ffff9d85b9d2f480 x1874531134284032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696662.163271:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134284032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696662.163282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696662.163291:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696662.163297:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134284032:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696662.163304:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134284160:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696662.163331:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696662.163341:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696662/real 1787696662] req@ffff9d85b9d2c000 x1874531134284160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696662.163349:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134284160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696662.163355:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696662.163357:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696662.163359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134284160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696662.163377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134284288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696662.163389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696662.163395:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696662/real 1787696662] req@ffff9d85b9d2ce00 x1874531134284288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696662.163415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134284288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696662.163421:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696662.163423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696662.163425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134284288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696662.163428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134284416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696662.163439:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696662.163462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696662/real 1787696662] req@ffff9d85b9d2e300 x1874531134284416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696662.163471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134284416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696662.163480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696662.163483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696662.163486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134284416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696662.163492:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134284544:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696662.163523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696662.163535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696662/real 1787696662] req@ffff9d85b9d2fb80 x1874531134284544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696662.163560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134284544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696662.163571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696662.163573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696662.163576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134284544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696662.163580:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134284672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696662.163595:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696662.163603:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696662/real 1787696662] req@ffff9d85b9d2ca80 x1874531134284672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696662.163610:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134284672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696662.163616:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696662.163618:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696662.163620:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134284672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696662.163623:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134284800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696662.163632:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696662.163638:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696662/real 1787696662] req@ffff9d85b9d2c380 x1874531134284800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696662.163644:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134284800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696662.163649:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696662.163651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696662.163654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134284800:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696662.739070:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696663.763014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696664.787014:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696665.811065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696666.835127:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696667.282969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696667.282984:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696667.282990:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696667.283005:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4569 00000100:00080000:3.0:1787696667.283017:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696667.283025:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696667.283080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134284928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696667.283121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696667.283124:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696667.283126:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696667.283130:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4569 00000100:00080000:3.0:1787696667.283133:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696667.283137:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696667.283147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696667.283162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134285056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696667.283189:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696667.283194:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696667.283196:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696667.283204:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4569 00000100:00080000:3.0:1787696667.283210:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696667.283215:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696667.283222:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696667.283237:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134285184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696667.283256:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696667.283261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696667.283264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696667.283270:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4569 00000100:00080000:3.0:1787696667.283273:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696667.283277:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696667.283281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696667.283292:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134285312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696667.283301:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696667.283309:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696667.283313:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696667.283321:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4569 00000100:00080000:3.0:1787696667.283324:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696667.283327:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696667.283332:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696667.283346:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134285440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696667.283373:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696667.283377:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696667.283378:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696667.283381:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4569 00000100:00080000:3.0:1787696667.283383:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696667.283386:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696667.283389:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696667.283395:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134285568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696667.283403:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696667.283408:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696667.283413:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696667.283416:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4569 00000100:00080000:3.0:1787696667.283418:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696667.283421:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696667.283424:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696667.283435:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134285696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696667.859124:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696668.882985:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696669.907080:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696670.930999:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696671.955106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696672.402961:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696672.402970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696672.402974:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696672.402982:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4574 00000100:00080000:3.0:1787696672.402991:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696672.402997:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696672.403031:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134285824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696672.403053:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696672.403056:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696672.403057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696672.403060:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4574 00000100:00080000:3.0:1787696672.403062:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696672.403065:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696672.403073:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696672.403082:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134285952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696672.403090:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696672.403093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696672.403093:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696672.403099:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4574 00000100:00080000:3.0:1787696672.403101:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696672.403107:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696672.403113:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696672.403120:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134286080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696672.403131:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696672.403133:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696672.403135:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696672.403137:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4574 00000100:00080000:3.0:1787696672.403140:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696672.403143:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696672.403145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696672.403154:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134286208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696672.403161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696672.403167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696672.403169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696672.403173:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4574 00000100:00080000:3.0:1787696672.403176:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696672.403180:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696672.403183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696672.403190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134286336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696672.403196:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696672.403198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696672.403199:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696672.403202:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4574 00000100:00080000:3.0:1787696672.403204:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696672.403208:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696672.403211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696672.403222:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134286464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696672.403257:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696672.403264:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696672.403267:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696672.403275:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4574 00000100:00080000:3.0:1787696672.403278:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696672.403284:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696672.403289:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696672.403301:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134286592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696672.979008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696674.003021:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696675.027055:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696676.051083:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696677.075102:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696677.522954:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696677.522965:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696677.522969:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696677.522979:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4579 00000100:00080000:3.0:1787696677.522990:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696677.522997:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696677.523033:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134286720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696677.523070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696677.523079:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696677.523082:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696677.523086:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4579 00000100:00080000:3.0:1787696677.523090:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696677.523095:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696677.523105:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696677.523119:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134286848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696677.523136:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696677.523140:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696677.523142:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696677.523147:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4579 00000100:00080000:3.0:1787696677.523150:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696677.523154:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696677.523159:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696677.523177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134286976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696677.523187:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696677.523191:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696677.523193:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696677.523200:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4579 00000100:00080000:3.0:1787696677.523207:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696677.523212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696677.523217:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696677.523228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134287104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696677.523238:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696677.523242:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696677.523244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696677.523249:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4579 00000100:00080000:3.0:1787696677.523253:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696677.523259:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696677.523268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696677.523281:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134287232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696677.523311:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696677.523316:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696677.523318:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696677.523323:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4579 00000100:00080000:3.0:1787696677.523328:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696677.523336:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696677.523342:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696677.523356:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134287360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696677.523397:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696677.523404:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696677.523408:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696677.523427:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4579 00000100:00080000:3.0:1787696677.523436:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696677.523445:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696677.523452:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696677.523469:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134287488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696678.099056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696679.123018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696680.147031:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696681.170973:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696682.195009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696682.643150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134287616:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696682.643217:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696682.643270:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696682/real 1787696682] req@ffff9d85b9d2e680 x1874531134287616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696682.643309:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134287616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696682.643339:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696682.643347:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696682.643355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134287616:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696682.643384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134287744:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696682.643413:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696682.643431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696682/real 1787696682] req@ffff9d85b9d2dc00 x1874531134287744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696682.643453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134287744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696682.643466:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696682.643472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696682.643479:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134287744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696682.643493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134287872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696682.643516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696682.643531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696682/real 1787696682] req@ffff9d85b9d2f800 x1874531134287872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696682.643551:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134287872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696682.643565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696682.643574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696682.643581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134287872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696682.643593:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134288000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696682.643613:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696682.643625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696682/real 1787696682] req@ffff9d85b9d2d500 x1874531134288000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696682.643644:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134288000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696682.643653:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696682.643656:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696682.643660:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134288000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696682.643691:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134288128:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696682.643710:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696682.643720:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696682/real 1787696682] req@ffff9d85b9d2ed80 x1874531134288128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696682.643733:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134288128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696682.643741:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696682.643745:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696682.643748:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134288128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696682.643754:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134288256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696682.643768:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696682.643777:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696682/real 1787696682] req@ffff9d85b9d2c700 x1874531134288256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696682.643788:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134288256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696682.643803:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696682.643806:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696682.643810:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134288256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696682.643815:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134288384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696682.643829:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696682.643837:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696682/real 1787696682] req@ffff9d85b9d2e680 x1874531134288384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696682.643867:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134288384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696682.643876:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696682.643880:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696682.643883:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134288384:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696683.219050:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696684.243022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696685.267046:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696686.291019:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696687.314999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696687.762990:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696687.763001:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696687.763006:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696687.763016:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4589 00000100:00080000:3.0:1787696687.763026:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696687.763032:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696687.763076:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134288512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696687.763113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696687.763116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696687.763117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696687.763121:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4589 00000100:00080000:3.0:1787696687.763125:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696687.763128:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696687.763137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696687.763154:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134288640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696687.763168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696687.763172:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696687.763174:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696687.763179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4589 00000100:00080000:3.0:1787696687.763185:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696687.763190:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696687.763197:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696687.763210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134288768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696687.763219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696687.763222:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696687.763223:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696687.763226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4589 00000100:00080000:3.0:1787696687.763230:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696687.763233:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696687.763237:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696687.763249:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134288896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696687.763260:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696687.763264:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696687.763267:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696687.763277:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4589 00000100:00080000:3.0:1787696687.763284:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696687.763292:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696687.763297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696687.763315:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134289024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696687.763337:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696687.763342:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696687.763352:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696687.763359:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4589 00000100:00080000:3.0:1787696687.763365:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696687.763372:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696687.763378:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696687.763395:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134289152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696687.763433:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696687.763441:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696687.763454:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696687.763463:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4589 00000100:00080000:3.0:1787696687.763469:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696687.763477:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696687.763484:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696687.763499:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134289280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696688.338987:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696689.363026:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696690.387019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696691.411021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696692.435035:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696692.882924:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696692.882935:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696692.882940:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696692.882951:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4594 00000100:00080000:3.0:1787696692.882964:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696692.882972:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696692.883026:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134289408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696692.883058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696692.883062:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696692.883063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696692.883068:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4594 00000100:00080000:3.0:1787696692.883071:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696692.883074:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696692.883084:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696692.883095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134289536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696692.883102:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696692.883104:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696692.883105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696692.883108:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4594 00000100:00080000:3.0:1787696692.883118:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696692.883121:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696692.883126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696692.883134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134289664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696692.883141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696692.883144:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696692.883146:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696692.883151:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4594 00000100:00080000:3.0:1787696692.883154:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696692.883160:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696692.883165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696692.883173:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134289792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696692.883181:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696692.883183:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696692.883185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696692.883187:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4594 00000100:00080000:3.0:1787696692.883189:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696692.883192:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696692.883195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696692.883201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134289920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696692.883208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696692.883211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696692.883212:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696692.883214:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4594 00000100:00080000:3.0:1787696692.883216:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696692.883219:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696692.883222:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696692.883229:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134290048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696692.883243:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696692.883246:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696692.883248:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696692.883251:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4594 00000100:00080000:3.0:1787696692.883255:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696692.883258:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696692.883261:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696692.883269:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134290176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696693.459073:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696694.483077:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696695.507084:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696696.530970:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696697.555076:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696698.002969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696698.002978:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696698.002981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696698.002990:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4599 00000100:00080000:3.0:1787696698.002999:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696698.003006:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696698.003042:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134290304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696698.003074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696698.003079:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696698.003081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696698.003085:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4599 00000100:00080000:3.0:1787696698.003089:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696698.003093:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696698.003101:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696698.003113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134290432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696698.003126:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696698.003129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696698.003130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696698.003134:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4599 00000100:00080000:3.0:1787696698.003137:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696698.003141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696698.003148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696698.003155:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134290560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696698.003161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696698.003164:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696698.003165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696698.003167:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4599 00000100:00080000:3.0:1787696698.003169:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696698.003171:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696698.003175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696698.003183:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134290688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696698.003189:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696698.003191:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696698.003192:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696698.003194:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4599 00000100:00080000:3.0:1787696698.003197:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696698.003200:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696698.003203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696698.003212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134290816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696698.003222:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696698.003224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696698.003226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696698.003229:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4599 00000100:00080000:3.0:1787696698.003231:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696698.003234:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696698.003236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696698.003243:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134290944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696698.003248:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696698.003250:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696698.003251:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696698.003253:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4599 00000100:00080000:3.0:1787696698.003256:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696698.003258:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696698.003261:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696698.003267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134291072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696699.603017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696700.626979:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696701.650994:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696702.675000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696703.122935:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696703.122945:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696703.122949:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696703.122959:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4604 00000100:00080000:3.0:1787696703.122971:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696703.122978:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696703.123017:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134291200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696703.123047:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696703.123051:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696703.123053:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696703.123058:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4604 00000100:00080000:3.0:1787696703.123062:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696703.123067:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696703.123076:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696703.123091:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134291328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696703.123107:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696703.123115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696703.123117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696703.123122:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4604 00000100:00080000:3.0:1787696703.123126:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696703.123131:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696703.123137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696703.123151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134291456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696703.123162:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696703.123169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696703.123174:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696703.123179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4604 00000100:00080000:3.0:1787696703.123184:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696703.123188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696703.123193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696703.123205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134291584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696703.123215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696703.123220:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696703.123225:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696703.123234:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4604 00000100:00080000:3.0:1787696703.123240:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696703.123246:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696703.123255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696703.123268:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134291712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696703.123292:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696703.123297:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696703.123302:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696703.123308:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4604 00000100:00080000:3.0:1787696703.123329:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696703.123332:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696703.123338:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696703.123354:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134291840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696703.123369:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696703.123373:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696703.123375:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696703.123380:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4604 00000100:00080000:3.0:1787696703.123386:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696703.123392:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696703.123400:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696703.123414:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134291968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696703.699040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696705.746998:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696706.771039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696707.794987:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696708.242935:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696708.242944:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696708.242948:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696708.242956:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4610 00000100:00080000:3.0:1787696708.242966:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696708.242971:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696708.243006:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134292096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696708.243027:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696708.243030:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696708.243031:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696708.243034:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4610 00000100:00080000:3.0:1787696708.243036:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696708.243038:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696708.243046:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696708.243054:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134292224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696708.243061:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696708.243063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696708.243065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696708.243068:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4610 00000100:00080000:3.0:1787696708.243070:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696708.243073:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696708.243076:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696708.243082:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134292352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696708.243088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696708.243092:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696708.243094:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696708.243100:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4610 00000100:00080000:3.0:1787696708.243106:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696708.243109:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696708.243111:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696708.243120:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134292480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696708.243126:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696708.243128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696708.243129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696708.243132:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4610 00000100:00080000:3.0:1787696708.243134:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696708.243137:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696708.243139:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696708.243146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134292608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696708.243151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696708.243154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696708.243155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696708.243158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4610 00000100:00080000:3.0:1787696708.243160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696708.243164:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696708.243170:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696708.243180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134292736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696708.243187:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696708.243189:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696708.243191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696708.243193:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4610 00000100:00080000:3.0:1787696708.243195:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696708.243197:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696708.243200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696708.243207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134292864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696708.819029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696709.842964:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696711.890979:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696712.915178:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696713.363104:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134292992:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696713.363228:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134293120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696713.363277:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134293248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696713.363302:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134293376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696713.363343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134293504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696713.363381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134293632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696713.363419:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134293760:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696713.939023:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696714.963054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696715.987032:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696716.435120:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696716.435172:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696716.435189:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696716.435237:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696716.435247:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696716.435255:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696716.435269:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696716.435276:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696716.435284:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787696716.435371:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696713/real 1787696716] req@ffff9d85b9d2f480 x1874531134293760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696716.435396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134293760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696716.435409:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696716.435413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696716.435417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134293760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696716.435426:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696713/real 1787696716] req@ffff9d85b9d2c000 x1874531134293632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696716.435439:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134293632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696716.435447:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696716.435450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696716.435453:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134293632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696716.435459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696713/real 1787696716] req@ffff9d85b9d2ce00 x1874531134293504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696716.435470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134293504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696716.435477:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696716.435480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696716.435483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134293504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696716.435489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696713/real 1787696716] req@ffff9d85b9d2e300 x1874531134293376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696716.435508:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134293376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696716.435517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696716.435521:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696716.435524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134293376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696716.435531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696713/real 1787696716] req@ffff9d85b9d2fb80 x1874531134293248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696716.435542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134293248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696716.435550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696716.435553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696716.435556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134293248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696716.435562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696713/real 1787696716] req@ffff9d85b9d2ca80 x1874531134293120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696716.435572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134293120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696716.435580:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696716.435583:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696716.435586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134293120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696716.435593:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696713/real 1787696716] req@ffff9d85b9d2c380 x1874531134292992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696716.435605:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134292992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696716.435612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696716.435615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696716.435618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134292992:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787696718.034999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696718.483103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134293888:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696718.483174:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696718.483241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696718/real 1787696718] req@ffff9d85b9d2c380 x1874531134293888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696718.483269:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134293888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696718.483290:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696718.483301:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696718.483308:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134293888:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696718.483343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696718.483379:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696718.483397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696718/real 1787696718] req@ffff9d85b9d2ca80 x1874531134294016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696718.483413:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134294016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696718.483421:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696718.483425:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696718.483428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696718.483435:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696718.483454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696718.483465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696718/real 1787696718] req@ffff9d85b9d2fb80 x1874531134294144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696718.483478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134294144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696718.483486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696718.483489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696718.483493:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696718.483499:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696718.483517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696718.483528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696718/real 1787696718] req@ffff9d85b9d2e300 x1874531134294272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696718.483538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134294272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696718.483545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696718.483548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696718.483552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696718.483558:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134294400:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696718.483577:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696718.483588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696718/real 1787696718] req@ffff9d85b9d2ce00 x1874531134294400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696718.483599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134294400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696718.483608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696718.483611:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696718.483614:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134294400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696718.483634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134294528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696718.483654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696718.483666:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696718/real 1787696718] req@ffff9d85b9d2c000 x1874531134294528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696718.483678:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134294528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696718.483687:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696718.483691:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696718.483694:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134294528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696718.483700:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134294656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696718.483719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696718.484068:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696718/real 1787696718] req@ffff9d85b9d2f480 x1874531134294656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696718.484084:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134294656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696718.484093:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696718.484098:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696718.484103:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134294656:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696719.059028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696720.083008:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696721.107020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696722.131000:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696723.603105:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134294784:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696723.603185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696723.603235:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696723/real 1787696723] req@ffff9d85b9d2e680 x1874531134294784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696723.603275:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134294784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696723.603285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696723.603288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696723.603292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134294784:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696723.603299:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294912:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696723.603324:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696723.603340:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696723/real 1787696723] req@ffff9d85b9d2c700 x1874531134294912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696723.603354:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134294912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696723.603363:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696723.603367:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696723.603371:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134294912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696723.603394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134295040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696723.603417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696723.603438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696723/real 1787696723] req@ffff9d85b9d2ed80 x1874531134295040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696723.603448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134295040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696723.603454:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696723.603456:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696723.603458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134295040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696723.603463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134295168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696723.603477:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696723.603485:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696723/real 1787696723] req@ffff9d85b9d2d500 x1874531134295168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696723.603492:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134295168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696723.603497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696723.603499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696723.603502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134295168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696723.603508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134295296:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696723.603521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696723.603528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696723/real 1787696723] req@ffff9d85b9d2f800 x1874531134295296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696723.603535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134295296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696723.603541:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696723.603542:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696723.603544:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134295296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696723.603548:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134295424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696723.603557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696723.603563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696723/real 1787696723] req@ffff9d85b9d2dc00 x1874531134295424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696723.603570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134295424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696723.603575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696723.603576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696723.603578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134295424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696723.603581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134295552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696723.603590:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696723.603596:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696723/real 1787696723] req@ffff9d85b9d2f480 x1874531134295552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696723.603603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134295552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696723.603608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696723.603609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696723.603611:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134295552:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696724.179045:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696725.203016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696726.227070:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696727.251000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696728.275049:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696728.723020:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696728.723031:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696728.723035:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696728.723046:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4630 00000100:00080000:3.0:1787696728.723056:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696728.723063:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696728.723101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134295680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696728.723131:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696728.723134:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696728.723136:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696728.723141:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4630 00000100:00080000:3.0:1787696728.723145:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696728.723150:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696728.723160:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696728.723179:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134295808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696728.723197:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696728.723200:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696728.723202:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696728.723206:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4630 00000100:00080000:3.0:1787696728.723210:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696728.723215:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696728.723221:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696728.723234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134295936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696728.723252:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696728.723256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696728.723258:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696728.723267:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4630 00000100:00080000:3.0:1787696728.723274:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696728.723279:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696728.723285:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696728.723301:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134296064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696728.723311:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696728.723315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696728.723317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696728.723322:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4630 00000100:00080000:3.0:1787696728.723326:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696728.723335:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696728.723345:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696728.723362:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134296192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696728.723391:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696728.723396:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696728.723401:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696728.723412:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4630 00000100:00080000:3.0:1787696728.723419:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696728.723428:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696728.723433:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696728.723445:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134296320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696728.723459:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696728.723464:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696728.723467:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696728.723472:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4630 00000100:00080000:3.0:1787696728.723477:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696728.723482:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696728.723487:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696728.723506:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134296448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696730.322966:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696731.346967:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696732.371018:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696733.395040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696733.843066:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134296576:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696733.843143:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696733.843216:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696733/real 1787696733] req@ffff9d85b9d2c000 x1874531134296576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696733.843243:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134296576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696733.843275:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696733.843286:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696733.843295:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134296576:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696733.843323:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134297344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696733.843325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134296704:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696733.843361:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696733.843391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696733/real 1787696733] req@ffff9d85b9d2ce00 x1874531134296704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696733.843405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134296704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696733.843414:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696733.843418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696733.843422:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134296704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696733.843428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134296832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696733.843451:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696733.843461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696733/real 1787696733] req@ffff9d85b9d2e300 x1874531134296832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696733.843470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134296832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696733.843475:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696733.843477:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696733.843480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134296832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696733.843484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134296960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696733.843495:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696733.843501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696733/real 1787696733] req@ffff9d85b9d2fb80 x1874531134296960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696733.843509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134296960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696733.843514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696733.843515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696733.843517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134296960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696733.843521:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696733.843531:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696733.843537:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696733/real 1787696733] req@ffff9d85b9d2ca80 x1874531134297088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696733.843549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134297088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696733.843554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696733.843556:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696733.843558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696733.843561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696733.843570:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696733.843576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696733/real 1787696733] req@ffff9d85b9d2c380 x1874531134297216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696733.843582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134297216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696733.843587:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696733.843588:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696733.843590:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696733.843604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696733.843614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696733.843620:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696733/real 1787696733] req@ffff9d85b9d2f480 x1874531134297344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696733.843626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134297344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696733.843635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696733.843637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696733.843639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297344:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696734.419544:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696735.442981:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696736.467004:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696737.491012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696738.514971:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696738.962954:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696738.962965:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696738.962970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696738.962980:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4640 00000100:00080000:1.0:1787696738.962991:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696738.962998:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696738.963047:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134297472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696738.963074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696738.963078:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696738.963080:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696738.963084:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4640 00000100:00080000:1.0:1787696738.963088:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696738.963092:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696738.963102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696738.963116:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134297600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696738.963124:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134297472:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696738.963127:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696738.963130:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696738.963132:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696738.963135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4640 00000100:00080000:1.0:1787696738.963138:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696738.963142:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696738.963147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696738.963158:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134297728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696738.963166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696738.963169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696738.963171:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696738.963174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4640 00000100:00080000:1.0:1787696738.963177:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696738.963180:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:3.0:1787696738.963184:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787696738.963185:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696738.963194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134297856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696738.963202:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696738.963205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696738.963206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696738.963209:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4640 00000100:00080000:1.0:1787696738.963212:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696738.963216:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696738.963220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696738.963230:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134297984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696738.963235:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696738/real 1787696738] req@ffff9d85b9d2dc00 x1874531134297472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696738.963245:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696738.963249:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696738.963251:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696738.963254:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134297472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696738.963255:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4640 00000100:00080000:1.0:1787696738.963258:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696738.963263:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787696738.963266:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787696738.963268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787696738.963270:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696738.963274:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134297472:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696738.963277:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134298112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696738.963285:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696738.963288:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696738.963290:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696738.963293:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4640 00000100:00080000:1.0:1787696738.963296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696738.963297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134297600:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696738.963299:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696738.963303:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696738.963313:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134298240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787696738.963318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696738.963331:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696738/real 1787696738] req@ffff9d85b9d2f800 x1874531134297600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696738.963344:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134297600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696738.963353:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696738.963358:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696738.963361:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134297600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696738.963368:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134297728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696738.963389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696738.963402:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696738/real 1787696738] req@ffff9d85b9d2d500 x1874531134297728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696738.963415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134297728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696738.963422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696738.963425:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696738.963427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134297728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696738.963431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134297856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696738.963446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696738.963453:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696738/real 1787696738] req@ffff9d85b9d2ed80 x1874531134297856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696738.963460:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134297856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696738.963465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696738.963467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696738.963468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134297856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696738.963472:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297984:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696738.963482:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696738.963489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696738/real 1787696738] req@ffff9d85b9d2c700 x1874531134297984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696738.963496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134297984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696738.963501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696738.963502:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696738.963504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134297984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696738.963507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134298112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696738.963516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696738.963521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696738/real 1787696738] req@ffff9d85b9d2e680 x1874531134298112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696738.963528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134298112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696738.963532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696738.963534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696738.963536:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134298112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696738.963548:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134298240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696738.963558:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696738.963564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696738/real 1787696738] req@ffff9d85b9d2dc00 x1874531134298240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696738.963575:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134298240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696738.963580:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696738.963582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696738.963584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134298240:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696739.538991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696740.563039:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696741.586923:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696742.610993:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696743.635002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696744.082958:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696744.082967:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696744.082970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696744.082978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4645 00000100:00080000:3.0:1787696744.082989:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696744.082995:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696744.083028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134298368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696744.083051:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696744.083054:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696744.083055:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696744.083058:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4645 00000100:00080000:3.0:1787696744.083060:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696744.083063:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696744.083071:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696744.083080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134298496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696744.083089:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696744.083093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696744.083095:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134298368:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696744.083095:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696744.083102:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4645 00000100:00080000:3.0:1787696744.083106:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696744.083108:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696744.083111:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696744.083121:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134298624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696744.083131:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696744.083135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696744.083137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696744.083140:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4645 00000100:00080000:3.0:1787696744.083145:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696744.083150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696744.083155:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696744.083172:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696744.083182:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134298752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696744.083191:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696744.083193:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696744.083195:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696744.083198:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4645 00000100:00080000:3.0:1787696744.083202:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696744.083205:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696744.083209:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696744.083223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134298880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696744.083224:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696744/real 1787696744] req@ffff9d85c0fb1c00 x1874531134298368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696744.083240:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696744.083247:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696744.083250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696744.083254:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4645 00000100:00080000:3.0:1787696744.083256:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696744.083261:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696744.083264:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134298368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696744.083265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696744.083276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134299008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696744.083279:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696744.083286:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696744.083288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696744.083292:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696744.083295:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696744.083296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134298368:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696744.083300:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4645 00000100:00080000:3.0:1787696744.083304:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696744.083306:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696744.083309:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696744.083315:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134299136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696744.083323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134298496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696744.083354:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696744.083365:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696744/real 1787696744] req@ffff9d85c0fb3800 x1874531134298496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696744.083374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134298496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696744.083379:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696744.083382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696744.083384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134298496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696744.083388:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134298624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696744.083398:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696744.083404:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696744/real 1787696744] req@ffff9d85c0fb2d80 x1874531134298624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696744.083412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134298624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696744.083417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696744.083418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696744.083420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134298624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696744.083424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134298752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696744.083434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696744.083439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696744/real 1787696744] req@ffff9d85c0fb3b80 x1874531134298752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696744.083447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134298752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696744.083452:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696744.083454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696744.083455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134298752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696744.083459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134298880:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696744.083468:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696744.083473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696744/real 1787696744] req@ffff9d85c0fb0380 x1874531134298880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696744.083488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134298880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696744.083493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696744.083495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696744.083497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134298880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696744.083501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696744.083512:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696744.083518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696744/real 1787696744] req@ffff9d85c0fb0a80 x1874531134299008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696744.083525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134299008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696744.083530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696744.083532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696744.083533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696744.083545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696744.083555:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696744.083560:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696744/real 1787696744] req@ffff9d85c0fb0e00 x1874531134299136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696744.083567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134299136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696744.083573:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696744.083574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696744.083576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299136:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696744.658998:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696745.683024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696746.706976:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696747.730956:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696748.755009:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696749.202962:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696749.202971:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696749.202975:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696749.202986:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4650 00000100:00080000:2.0:1787696749.202997:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696749.203006:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696749.203042:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134299264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696749.203072:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696749.203076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696749.203078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696749.203083:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4650 00000100:00080000:2.0:1787696749.203087:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696749.203091:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696749.203101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696749.203116:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134299392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696749.203123:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134299264:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787696749.203126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696749.203129:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696749.203131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696749.203135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4650 00000100:00080000:2.0:1787696749.203138:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696749.203141:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696749.203146:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696749.203156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134299520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696749.203164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696749.203167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696749.203168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696749.203171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4650 00000100:00080000:2.0:1787696749.203174:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696749.203177:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787696749.203181:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787696749.203181:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696749.203190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134299648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696749.203204:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696749.203207:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696749.203208:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696749.203211:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4650 00000100:00080000:2.0:1787696749.203214:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696749.203218:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696749.203220:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696749/real 1787696749] req@ffff9d85babd6a00 x1874531134299264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696749.203222:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696749.203230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134299776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696749.203237:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134299264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696749.203238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696749.203242:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696749.203244:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696749.203247:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4650 00000100:00080000:1.0:1787696749.203248:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696749.203250:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696749.203252:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787696749.203254:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696749.203257:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134299264:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787696749.203258:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696749.203266:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134299904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696749.203274:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696749.203277:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696749.203279:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134299392:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787696749.203279:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696749.203283:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4650 00000100:00080000:2.0:1787696749.203286:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696749.203290:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696749.203294:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696749.203299:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787696749.203303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134300032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696749.203310:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696749/real 1787696749] req@ffff9d85babd6680 x1874531134299392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696749.203322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134299392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696749.203331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696749.203335:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696749.203339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134299392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696749.203346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134299520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696749.203364:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696749.203372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696749/real 1787696749] req@ffff9d85babd4000 x1874531134299520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696749.203380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134299520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696749.203385:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696749.203387:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696749.203389:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134299520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696749.203393:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134299648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696749.203404:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696749.203410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696749/real 1787696749] req@ffff9d85babd4380 x1874531134299648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696749.203416:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134299648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696749.203421:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696749.203423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696749.203425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134299648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696749.203428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299776:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696749.203438:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696749.203444:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696749/real 1787696749] req@ffff9d85babd6300 x1874531134299776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696749.203451:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134299776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696749.203456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696749.203458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696749.203459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696749.203463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696749.203479:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696749.203489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696749/real 1787696749] req@ffff9d85babd7480 x1874531134299904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696749.203500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134299904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696749.203508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696749.203512:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696749.203546:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134299904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696749.203563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696749.203582:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696749.203593:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696749/real 1787696749] req@ffff9d85babd5c00 x1874531134300032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696749.203604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134300032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696749.203611:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696749.203615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696749.203618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300032:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696749.779030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696750.802994:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696751.827004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696752.850971:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696753.874967:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696754.322970:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696754.322981:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696754.322985:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696754.322996:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4656 00000100:00080000:1.0:1787696754.323007:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696754.323015:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696754.323055:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134300160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696754.323088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696754.323091:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696754.323094:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696754.323098:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4656 00000100:00080000:1.0:1787696754.323102:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696754.323106:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696754.323114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696754.323126:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134300288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696754.323139:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134300160:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696754.323140:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696754.323148:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696754.323154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696754.323163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4656 00000100:00080000:1.0:1787696754.323165:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696754.323167:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696754.323171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696754.323184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134300416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696754.323194:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696754.323198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696754.323200:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696754.323206:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4656 00000100:00080000:1.0:1787696754.323211:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696754.323216:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696754.323229:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696754.323239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134300544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696754.323246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696754.323252:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787696754.323255:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696754.323257:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696754.323265:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4656 00000100:00080000:1.0:1787696754.323277:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696754.323284:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696754.323290:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696754.323305:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134300672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696754.323313:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696754.323315:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696754.323317:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696754.323321:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4656 00000100:00080000:1.0:1787696754.323323:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696754.323326:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696754.323330:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696754.323339:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134300800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696754.323347:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787696754.323351:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696754/real 1787696754] req@ffff9d85b9d2f480 x1874531134300160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.323352:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696754.323357:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696754.323369:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4656 00000100:00080000:1.0:1787696754.323380:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696754.323388:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696754.323393:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696754.323401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134300160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.323412:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134300928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696754.323418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696754.323448:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696754.323455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134300160:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696754.323487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134300288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696754.323517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696754.323533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696754/real 1787696754] req@ffff9d85b9d2c380 x1874531134300288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696754.323548:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134300288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696754.323568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696754.323572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696754.323576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134300288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696754.323586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134300416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696754.323612:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696754.323626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696754/real 1787696754] req@ffff9d85b9d2ca80 x1874531134300416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696754.323640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134300416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696754.323649:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696754.323653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696754.323658:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134300416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696754.323665:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134300544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696754.323700:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696754.323713:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696754/real 1787696754] req@ffff9d85b9d2fb80 x1874531134300544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696754.323727:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134300544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696754.323736:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696754.323740:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696754.323744:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134300544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696754.323751:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300672:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696754.323775:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696754.323787:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696754/real 1787696754] req@ffff9d85b9d2e300 x1874531134300672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696754.323800:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134300672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696754.323808:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696754.323812:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696754.323815:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696754.323823:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696754.323844:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696754.323884:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696754/real 1787696754] req@ffff9d85b9d2ce00 x1874531134300800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696754.323909:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134300800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696754.323919:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696754.323922:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696754.323926:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696754.323933:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696754.323954:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696754.323967:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696754/real 1787696754] req@ffff9d85b9d2c000 x1874531134300928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696754.323978:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134300928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696754.323994:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696754.323998:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696754.324001:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134300928:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696754.899026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696755.923125:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696756.947054:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696757.971103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696758.995044:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696759.442990:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696759.442998:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696759.443002:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696759.443010:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4661 00000100:00080000:3.0:1787696759.443019:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696759.443025:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696759.443059:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134301056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696759.443083:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696759.443086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696759.443087:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696759.443090:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4661 00000100:00080000:3.0:1787696759.443093:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696759.443095:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696759.443103:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696759.443112:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134301184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696759.443119:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696759.443123:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696759.443124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696759.443128:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4661 00000100:00080000:3.0:1787696759.443130:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696759.443132:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696759.443135:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696759.443144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134301312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696759.443154:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134301056:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696759.443155:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696759.443169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696759.443173:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696759.443182:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4661 00000100:00080000:3.0:1787696759.443185:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696759.443192:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696759.443195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696759.443204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134301440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696759.443211:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696759.443213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696759.443215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696759.443218:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4661 00000100:00080000:3.0:1787696759.443221:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696759.443223:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696759.443228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696759.443235:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134301568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696759.443241:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696759.443243:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696759.443244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696759.443247:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4661 00000100:00080000:3.0:1787696759.443249:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696759.443251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696759.443254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696759.443264:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134301696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696759.443266:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696759.443278:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696759.443282:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696759.443283:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696759.443286:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4661 00000100:00080000:3.0:1787696759.443289:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696759.443295:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696759.443299:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696759.443308:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134301824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696759.443363:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696759/real 1787696759] req@ffff9d85c0fb2a00 x1874531134301056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696759.443384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134301056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696759.443393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696759.443397:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696759.443402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134301056:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696759.443409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134301184:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696759.443431:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696759.443442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696759/real 1787696759] req@ffff9d85c0fb2300 x1874531134301184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696759.443456:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134301184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696759.443465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696759.443469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696759.443473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134301184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696759.443510:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134301312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696759.443529:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696759.443537:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696759/real 1787696759] req@ffff9d85c0fb1500 x1874531134301312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696759.443546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134301312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696759.443551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696759.443554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696759.443556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134301312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696759.443560:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134301440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696759.443571:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696759.443580:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696759/real 1787696759] req@ffff9d85c0fb2680 x1874531134301440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696759.443592:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134301440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696759.443617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696759.443621:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696759.443625:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134301440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696759.443632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134301568:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696759.443653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696759.443667:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696759/real 1787696759] req@ffff9d85c0fb3100 x1874531134301568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696759.443679:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134301568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696759.443695:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696759.443699:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696759.443703:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134301568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696759.443709:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134301696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696759.443738:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696759.443749:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696759/real 1787696759] req@ffff9d85c0fb3480 x1874531134301696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696759.443761:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134301696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696759.443768:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696759.443772:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696759.443776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134301696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696759.443783:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134301824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696759.443803:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696759.443814:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696759/real 1787696759] req@ffff9d85c0fb1880 x1874531134301824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696759.443826:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134301824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696759.443833:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696759.443835:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696759.443837:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134301824:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787696760.019049:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696761.042995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696762.067048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696763.090979:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696764.114970:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696764.562933:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696764.562943:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696764.562948:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696764.562958:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4666 00000100:00080000:3.0:1787696764.562969:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696764.562986:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696764.563029:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134301952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696764.563058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696764.563063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696764.563065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696764.563070:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4666 00000100:00080000:3.0:1787696764.563074:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696764.563078:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696764.563088:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696764.563103:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134302080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696764.563116:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696764.563121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134301952:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696764.563122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696764.563126:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696764.563135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4666 00000100:00080000:3.0:1787696764.563139:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696764.563144:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696764.563151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696764.563167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134302208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696764.563177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696764.563180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696764.563183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696764.563187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4666 00000100:00080000:3.0:1787696764.563192:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696764.563196:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696764.563201:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696764.563208:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696764.563223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134302336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696764.563233:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696764.563236:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696764.563238:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696764.563242:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4666 00000100:00080000:3.0:1787696764.563247:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696764.563261:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696764/real 1787696764] req@ffff9d85c0fb1f80 x1874531134301952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696764.563270:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696764.563278:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696764.563284:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134301952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696764.563294:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696764.563298:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134302464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696764.563300:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696764.563308:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134301952:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696764.563321:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696764.563326:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696764.563328:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696764.563333:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4666 00000100:00080000:3.0:1787696764.563338:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696764.563342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302080:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696764.563345:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696764.563355:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696764.563371:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134302592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696764.563380:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696764.563383:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696764.563389:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696764.563391:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696764.563395:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696764/real 1787696764] req@ffff9d85c0fb0000 x1874531134302080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696764.563399:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4666 00000100:00080000:3.0:1787696764.563409:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696764.563414:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134302080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696764.563418:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696764.563429:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696764.563430:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696764.563437:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696764.563439:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696764.563449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696764.563450:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134302720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696764.563466:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696764.563475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696764/real 1787696764] req@ffff9d85c0fb0700 x1874531134302208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696764.563484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134302208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696764.563490:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696764.563493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696764.563495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696764.563498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696764.563508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696764.563514:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696764/real 1787696764] req@ffff9d85c0fb1880 x1874531134302336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696764.563525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134302336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696764.563531:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696764.563533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696764.563534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696764.563538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134302464:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696764.563547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696764.563552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696764/real 1787696764] req@ffff9d85c0fb3480 x1874531134302464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696764.563559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134302464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696764.563564:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696764.563566:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696764.563568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134302464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696764.563579:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134302592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696764.563588:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696764.563594:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696764/real 1787696764] req@ffff9d85c0fb3100 x1874531134302592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696764.563604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134302592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696764.563619:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696764.563622:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696764.563626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134302592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696764.563633:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134302720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696764.563653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696764.563662:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696764/real 1787696764] req@ffff9d85c0fb2680 x1874531134302720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696764.563670:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134302720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696764.563675:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696764.563677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696764.563679:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134302720:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696765.139021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696766.162948:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696767.186997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696768.211019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696769.235003:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696769.682982:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696769.682992:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696769.682997:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696769.683007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4671 00000100:00080000:1.0:1787696769.683018:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696769.683026:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696769.683074:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134302848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696769.683103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696769.683107:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696769.683109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696769.683113:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4671 00000100:00080000:1.0:1787696769.683118:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696769.683123:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696769.683133:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696769.683141:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134302848:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696769.683150:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134302976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696769.683176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696769.683180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696769.683182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696769.683187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4671 00000100:00080000:1.0:1787696769.683193:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696769.683197:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696769.683203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696769.683216:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696769.683219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134303104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696769.683231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696769.683236:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696769.683238:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696769.683242:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4671 00000100:00080000:1.0:1787696769.683246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696769.683250:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696769.683257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696769.683264:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696769/real 1787696769] req@ffff9d85b9d2c000 x1874531134302848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696769.683279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134303232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696769.683285:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134302848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696769.683298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696769.683309:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696769.683313:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696769.683317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696769.683321:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4671 00000100:00080000:3.0:1787696769.683324:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787696769.683329:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696769.683331:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134302848:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696769.683341:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696769.683348:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696769.683356:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696769.683363:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134303360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696769.683376:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787696769.683379:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696769.683385:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696769.683388:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696769.683395:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696769/real 1787696769] req@ffff9d85b9d2ce00 x1874531134302976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696769.683396:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4671 00000100:00080000:1.0:1787696769.683404:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696769.683410:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696769.683415:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696769.683422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134302976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696769.683431:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134303488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696769.683431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696769.683438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696769.683444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134302976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696769.683454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134303104:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787696769.683456:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696769.683461:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696769.683465:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696769.683470:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4671 00000800:00000100:3.0:1787696769.683475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696769.683478:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696769.683486:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696769.683487:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696769/real 1787696769] req@ffff9d85b9d2e300 x1874531134303104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787696769.683496:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696769.683500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134303104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696769.683509:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696769.683512:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696769.683514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134303104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696769.683519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134303232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696769.683523:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134303616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787696769.683536:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696769.683546:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696769/real 1787696769] req@ffff9d85b9d2fb80 x1874531134303232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696769.683554:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134303232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696769.683561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696769.683564:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696769.683566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134303232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696769.683585:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134303360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696769.683597:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696769.683604:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696769/real 1787696769] req@ffff9d85b9d2ca80 x1874531134303360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696769.683612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134303360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696769.683617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696769.683619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696769.683621:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134303360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696769.683625:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134303488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696769.683634:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696769.683640:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696769/real 1787696769] req@ffff9d85b9d2c380 x1874531134303488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696769.683647:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134303488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696769.683652:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696769.683654:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696769.683656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134303488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696769.683659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134303616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696769.683668:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696769.683674:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696769/real 1787696769] req@ffff9d85b9d2f480 x1874531134303616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696769.683681:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134303616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696769.683686:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696769.683687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696769.683689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134303616:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696770.259106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696771.283052:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696772.306981:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696773.331003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696774.355049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696774.803010:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696774.803023:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696774.803027:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696774.803038:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4676 00000100:00080000:3.0:1787696774.803051:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696774.803058:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696774.803153:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134303744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696774.803189:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696774.803194:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696774.803196:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696774.803202:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4676 00000100:00080000:3.0:1787696774.803206:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696774.803212:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696774.803223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696774.803239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134303872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696774.803253:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696774.803261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696774.803262:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134303744:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696774.803264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696774.803281:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4676 00000100:00080000:3.0:1787696774.803285:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696774.803291:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696774.803297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696774.803316:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134304000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696774.803326:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696774.803331:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696774.803333:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696774.803339:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4676 00000800:00000100:1.0:1787696774.803346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696774.803350:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696774.803359:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696774.803365:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696774.803379:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134304128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696774.803387:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696774.803390:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696774.803392:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696774.803397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696774/real 1787696774] req@ffff9d85c0fb1500 x1874531134303744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696774.803397:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4676 00000100:00080000:3.0:1787696774.803407:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696774.803422:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696774.803431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134303744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696774.803432:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696774.803452:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696774.803455:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696774.803459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134303744:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696774.803470:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134304256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696774.803482:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696774.803483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134303872:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696774.803498:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696774.803501:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696774.803506:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4676 00000800:00000100:1.0:1787696774.803514:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696774.803517:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696774.803526:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696774.803530:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696774/real 1787696774] req@ffff9d85c0fb2300 x1874531134303872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696774.803536:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696774.803542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134303872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696774.803554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696774.803557:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134304384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696774.803559:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696774.803569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134303872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696774.803580:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304000:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787696774.803602:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787696774.803608:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696774.803616:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696774.803619:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696774.803622:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696774/real 1787696774] req@ffff9d85c0fb2a00 x1874531134304000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696774.803631:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4676 00000100:00080000:3.0:1787696774.803644:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696774.803651:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134304000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696774.803657:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696774.803666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696774.803673:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787696774.803674:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696774.803684:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696774.803694:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696774.803697:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134304512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696774.803722:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696774.803730:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696774/real 1787696774] req@ffff9d85c0fb0e00 x1874531134304128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696774.803739:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134304128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696774.803745:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696774.803748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696774.803752:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696774.803763:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134304256:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696774.803774:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696774.803780:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696774/real 1787696774] req@ffff9d85c0fb0a80 x1874531134304256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696774.803787:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134304256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696774.803791:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696774.803793:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696774.803795:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134304256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696774.803798:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134304384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696774.803808:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696774.803814:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696774/real 1787696774] req@ffff9d85c0fb0380 x1874531134304384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696774.803821:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134304384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696774.803826:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696774.803828:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696774.803830:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134304384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696774.803833:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134304512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696774.803842:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696774.803848:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696774/real 1787696774] req@ffff9d85c0fb3b80 x1874531134304512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696774.803876:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134304512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696774.803881:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696774.803883:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696774.803885:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134304512:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696775.379040:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696776.403040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696777.427011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696778.450958:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696779.475037:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696779.923020:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696779.923030:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696779.923033:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696779.923042:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4681 00000100:00080000:3.0:1787696779.923051:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696779.923057:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696779.923091:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134304640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696779.923114:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696779.923117:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696779.923118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696779.923121:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4681 00000100:00080000:3.0:1787696779.923124:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696779.923127:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696779.923134:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696779.923143:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134304768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696779.923150:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696779.923153:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696779.923155:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696779.923157:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134304640:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696779.923160:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4681 00000100:00080000:3.0:1787696779.923165:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696779.923168:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696779.923171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696779.923180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134304896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696779.923187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696779.923189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696779.923190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696779.923193:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4681 00000100:00080000:3.0:1787696779.923195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696779.923197:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696779.923200:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696779.923208:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134305024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696779.923215:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696779.923217:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696779.923219:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696779.923221:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4681 00000100:00080000:3.0:1787696779.923224:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696779.923226:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696779.923229:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696779.923236:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134305152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696779.923243:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696779.923246:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696779.923249:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696779.923254:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4681 00000100:00080000:3.0:1787696779.923257:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696779.923262:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696779.923266:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304768:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787696779.923270:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696779.923287:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134305280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696779.923299:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696779.923302:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696779.923303:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696779.923306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304896:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787696779.923309:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4681 00000100:00080000:3.0:1787696779.923316:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696779.923319:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696779.923325:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696779.923332:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696779.923337:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134305408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696779.923352:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134305152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696779.923372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134305280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696779.923387:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134305408:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696780.499025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696781.523012:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696782.547004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787696782.995157:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:1.0:1787696782.995216:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787696782.995234:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787696782.995276:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787696782.995285:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787696782.995293:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787696782.995301:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787696782.995310:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787696782.995320:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787696782.995389:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696779/real 1787696782] req@ffff9d85c0fb0a80 x1874531134305280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696782.995432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134305280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.995448:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.995455:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696782.995459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134305280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696782.995470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696779/real 1787696782] req@ffff9d85c0fb0e00 x1874531134305408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696782.995486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134305408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.995494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.995498:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696782.995503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134305408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696782.995511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696779/real 1787696782] req@ffff9d85c0fb3800 x1874531134304768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696782.995524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134304768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.995533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.995537:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696782.995541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696782.995548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696779/real 1787696782] req@ffff9d85c0fb1c00 x1874531134304896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696782.995560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134304896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.995568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.995572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696782.995575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134304896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696782.995583:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696779/real 1787696782] req@ffff9d85c0fb3b80 x1874531134305024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696782.995595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134305024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.995604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.995607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696782.995611:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696782.995619:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696779/real 1787696782] req@ffff9d85c0fb0380 x1874531134305152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696782.995632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134305152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.995640:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.995644:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696782.995648:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134305152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696782.995655:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696779/real 1787696782] req@ffff9d85c0fb2d80 x1874531134304640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696782.995667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134304640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.995675:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.995679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696782.995681:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134304640:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787696783.571015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696784.595029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696785.043002:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696785.043014:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696785.043018:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696785.043028:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4686 00000100:00080000:1.0:1787696785.043040:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696785.043048:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696785.043081:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134305536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696785.043109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696785.043112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696785.043115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696785.043120:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4686 00000100:00080000:1.0:1787696785.043124:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696785.043129:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696785.043137:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134305536:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787696785.043139:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696785.043164:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134305664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696785.043176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696785.043179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696785.043182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696785.043186:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4686 00000100:00080000:1.0:1787696785.043192:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696785.043197:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696785.043203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696785.043209:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696785.043218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134305792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696785.043229:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696785.043233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696785.043235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696785.043239:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4686 00000100:00080000:1.0:1787696785.043243:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696785.043247:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696785.043251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696785.043263:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696785/real 1787696785] req@ffff9d85b9d2dc00 x1874531134305536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696785.043266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134305920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696785.043296:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696785.043300:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696785.043303:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696785.043308:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134305536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696785.043310:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4686 00000100:00080000:1.0:1787696785.043315:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696785.043321:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696785.043329:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787696785.043337:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787696785.043344:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134306048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696785.043345:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696785.043354:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134305536:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696785.043366:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696785.043370:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696785.043372:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696785.043378:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4686 00000100:00080000:1.0:1787696785.043382:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696785.043386:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696785.043392:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696785.043393:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696785.043406:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134306176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696785.043418:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696785.043422:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696785.043424:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696785.043431:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4686 00000100:00080000:1.0:1787696785.043435:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787696785.043440:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696785.043442:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696785.043450:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696785.043462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696785/real 1787696785] req@ffff9d85b9d2e680 x1874531134305664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696785.043465:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134306304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696785.043502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134305664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696785.043512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696785.043516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696785.043519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696785.043526:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696785.043563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696785.043576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696785/real 1787696785] req@ffff9d85b9d2c700 x1874531134305792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696785.043587:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134305792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696785.043594:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696785.043598:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696785.043600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696785.043606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696785.043625:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696785.043635:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696785/real 1787696785] req@ffff9d85b9d2ed80 x1874531134305920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696785.043653:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134305920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696785.043661:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696785.043664:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696785.043668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134305920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696785.043673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306048:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696785.043692:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696785.043704:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696785/real 1787696785] req@ffff9d85b9d2d500 x1874531134306048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696785.043717:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134306048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696785.043799:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696785.043804:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696785.043814:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306048:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696785.043834:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696785.043881:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696785.043920:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696785/real 1787696785] req@ffff9d85b9d2f800 x1874531134306176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696785.043934:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134306176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696785.043949:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696785.043953:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696785.043956:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696785.043962:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696785.043985:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696785.043998:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696785/real 1787696785] req@ffff9d85b9d2dc00 x1874531134306304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696785.044010:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134306304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696785.044020:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696785.044023:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696785.044027:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306304:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696785.619003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696786.643015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696787.666986:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696788.691015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696789.714991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696790.162915:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696790.162926:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696790.162931:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696790.162941:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4691 00000100:00080000:1.0:1787696790.162953:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696790.162961:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696790.162998:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134306432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696790.163021:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696790.163023:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696790.163024:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696790.163027:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4691 00000100:00080000:1.0:1787696790.163030:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696790.163033:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696790.163041:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696790.163050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134306560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696790.163058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696790.163062:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696790.163064:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696790.163064:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134306432:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696790.163071:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4691 00000100:00080000:1.0:1787696790.163075:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696790.163078:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696790.163082:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696790.163092:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134306688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696790.163098:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696790.163100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696790.163102:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696790.163104:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4691 00000100:00080000:1.0:1787696790.163107:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696790.163109:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696790.163112:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696790.163119:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134306816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696790.163125:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696790.163127:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696790.163128:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696790.163131:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4691 00000100:00080000:1.0:1787696790.163134:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787696790.163138:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696790.163141:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696790.163145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696790.163152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134306944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696790.163159:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696790.163161:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696790.163163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696790.163165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4691 00000100:00080000:1.0:1787696790.163167:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696790.163170:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696790.163174:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696790.163180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134307072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696790.163185:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696790.163188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696790.163189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696790.163192:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4691 00000100:00080000:1.0:1787696790.163193:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696790.163196:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696790.163201:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696790.163202:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696790/real 1787696790] req@ffff9d85b9d2f480 x1874531134306432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696790.163212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134307200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696790.163227:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134306432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696790.163236:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696790.163241:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696790.163244:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134306432:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696790.163272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134306560:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696790.163299:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696790.163314:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696790/real 1787696790] req@ffff9d85b9d2c380 x1874531134306560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696790.163324:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134306560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696790.163330:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696790.163332:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696790.163335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134306560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696790.163339:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134306688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696790.163354:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696790.163362:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696790/real 1787696790] req@ffff9d85b9d2ca80 x1874531134306688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696790.163369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134306688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696790.163375:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696790.163377:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696790.163379:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134306688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696790.163382:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134306816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696790.163393:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696790.163399:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696790/real 1787696790] req@ffff9d85b9d2fb80 x1874531134306816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696790.163406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134306816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696790.163411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696790.163413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696790.163415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134306816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696790.163418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306944:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696790.163428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696790.163434:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696790/real 1787696790] req@ffff9d85b9d2e300 x1874531134306944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696790.163441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134306944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696790.163446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696790.163448:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696790.163450:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134306944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696790.163453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696790.163462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696790.163467:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696790/real 1787696790] req@ffff9d85b9d2ce00 x1874531134307072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696790.163488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134307072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696790.163494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696790.163496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696790.163498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696790.163501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696790.163511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696790.163517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696790/real 1787696790] req@ffff9d85b9d2c000 x1874531134307200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696790.163524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134307200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696790.163530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696790.163531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696790.163533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307200:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787696790.738951:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696791.762966:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696792.787041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696793.811040:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696794.834991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696795.282971:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696795.282980:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696795.282983:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696795.282991:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4697 00000100:00080000:3.0:1787696795.283001:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696795.283007:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696795.283041:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134307328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696795.283064:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696795.283067:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696795.283068:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696795.283072:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4697 00000100:00080000:3.0:1787696795.283074:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696795.283077:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696795.283084:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696795.283093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134307456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696795.283101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134307328:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696795.283103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696795.283109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696795.283111:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696795.283114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4697 00000100:00080000:3.0:1787696795.283116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696795.283118:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696795.283122:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696795.283130:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134307584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696795.283136:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696795.283139:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696795.283140:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696795.283143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4697 00000100:00080000:3.0:1787696795.283146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696795.283148:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696795.283151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696795.283157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134307712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696795.283162:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696795.283165:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696795.283168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696795.283170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696795.283173:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4697 00000100:00080000:3.0:1787696795.283176:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696795.283179:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696795.283182:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696795.283190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134307840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696795.283195:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696795.283197:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696795.283198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696795.283201:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4697 00000100:00080000:3.0:1787696795.283203:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696795.283206:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696795.283209:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696795.283217:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696795/real 1787696795] req@ffff9d85c0fb2d80 x1874531134307328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696795.283218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134307968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696795.283245:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696795.283248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696795.283249:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696795.283253:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134307328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696795.283265:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4697 00000100:00080000:3.0:1787696795.283269:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696795.283270:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696795.283273:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696795.283277:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787696795.283279:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696795.283283:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134307328:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696795.283290:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134308096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696795.283306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134307456:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696795.283326:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696795.283335:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696795/real 1787696795] req@ffff9d85c0fb0380 x1874531134307456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696795.283343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134307456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696795.283349:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696795.283351:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696795.283353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134307456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696795.283357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134307584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696795.283367:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696795.283373:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696795/real 1787696795] req@ffff9d85c0fb3b80 x1874531134307584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696795.283381:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134307584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696795.283390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696795.283393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696795.283396:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134307584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696795.283402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134307712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696795.283423:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696795.283435:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696795/real 1787696795] req@ffff9d85c0fb1c00 x1874531134307712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696795.283450:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134307712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696795.283455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696795.283457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696795.283459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134307712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696795.283463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307840:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696795.283476:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696795.283483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696795/real 1787696795] req@ffff9d85c0fb3800 x1874531134307840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696795.283490:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134307840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696795.283495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696795.283497:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696795.283498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696795.283502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696795.283511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696795.283516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696795/real 1787696795] req@ffff9d85c0fb0e00 x1874531134307968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696795.283523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134307968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696795.283532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696795.283534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696795.283536:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134307968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696795.283539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696795.283548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696795.283553:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696795/real 1787696795] req@ffff9d85c0fb0a80 x1874531134308096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696795.283560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134308096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696795.283565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696795.283566:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696795.283568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308096:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696795.859021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696796.882951:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696797.907002:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696798.930998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696799.955059:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696800.402909:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696800.402920:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696800.402924:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696800.402935:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4702 00000100:00080000:3.0:1787696800.402948:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696800.402956:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696800.402992:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134308224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696800.403028:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696800.403032:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696800.403034:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696800.403039:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4702 00000100:00080000:3.0:1787696800.403044:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696800.403049:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696800.403059:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696800.403075:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134308352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696800.403081:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134308224:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696800.403102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696800.403106:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696800.403108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696800.403114:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4702 00000100:00080000:3.0:1787696800.403118:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696800.403123:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696800.403130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696800.403147:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134308480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696800.403148:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696800.403162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696800.403165:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696800.403167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696800.403170:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4702 00000100:00080000:3.0:1787696800.403175:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696800.403179:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696800.403184:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696800.403193:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134308608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696800.403203:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696800/real 1787696800] req@ffff9d85c0fb2a00 x1874531134308224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696800.403220:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696800.403225:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696800.403227:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696800.403232:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4702 00000100:00080000:3.0:1787696800.403235:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696800.403237:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134308224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696800.403243:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696800.403253:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696800.403258:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696800.403267:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696800.403267:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134308736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696800.403277:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134308224:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696800.403284:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696800.403290:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696800.403292:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696800.403297:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4702 00000100:00080000:3.0:1787696800.403302:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696800.403307:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134308352:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696800.403309:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696800.403320:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696800.403336:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134308864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696800.403343:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696800.403354:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696800.403359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696800/real 1787696800] req@ffff9d85c0fb2300 x1874531134308352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696800.403361:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696800.403365:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696800.403376:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4702 00000100:00100000:1.0:1787696800.403379:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134308352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696800.403383:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696800.403393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696800.403393:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696800.403401:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787696800.403403:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696800.403404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134308352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696800.403415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134308480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696800.403425:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134308992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696800.403438:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696800.403450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696800/real 1787696800] req@ffff9d85c0fb1500 x1874531134308480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696800.403458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134308480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696800.403463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696800.403465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696800.403467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134308480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696800.403471:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134308608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696800.403482:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696800.403488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696800/real 1787696800] req@ffff9d85c0fb2680 x1874531134308608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696800.403496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134308608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696800.403501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696800.403502:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696800.403504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134308608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696800.403507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308736:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696800.403517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696800.403523:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696800/real 1787696800] req@ffff9d85c0fb3100 x1874531134308736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696800.403530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134308736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696800.403535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696800.403536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696800.403538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696800.403549:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696800.403559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696800.403564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696800/real 1787696800] req@ffff9d85c0fb3480 x1874531134308864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696800.403571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134308864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696800.403577:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696800.403579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696800.403582:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696800.403585:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696800.403593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696800.403598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696800/real 1787696800] req@ffff9d85c0fb1880 x1874531134308992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696800.403605:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134308992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696800.403610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696800.403612:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696800.403614:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134308992:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696800.979035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696802.002980:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696803.026977:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696804.051065:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696805.075008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696805.522966:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696805.522993:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696805.522998:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696805.523007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4707 00000100:00080000:1.0:1787696805.523019:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696805.523026:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696805.523065:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134309120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696805.523092:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696805.523096:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696805.523098:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696805.523102:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4707 00000100:00080000:1.0:1787696805.523107:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696805.523118:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696805.523128:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696805.523128:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134309120:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696805.523146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134309248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696805.523160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696805.523163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696805.523165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696805.523170:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4707 00000100:00080000:1.0:1787696805.523174:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696805.523178:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696805.523184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696805.523194:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696805.523199:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134309376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696805.523212:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696805.523216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696805.523217:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696805.523222:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4707 00000100:00080000:1.0:1787696805.523225:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696805.523230:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696805.523236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696805.523240:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696805/real 1787696805] req@ffff9d85b9d2c000 x1874531134309120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696805.523251:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134309504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696805.523264:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787696805.523266:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134309120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696805.523269:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696805.523274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696805.523284:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4707 00000100:00080000:1.0:1787696805.523287:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696805.523292:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787696805.523297:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787696805.523299:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787696805.523306:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696805.523312:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134309120:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696805.523317:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134309632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696805.523328:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696805.523332:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696805.523335:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696805.523339:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134309248:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696805.523343:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4707 00000100:00080000:1.0:1787696805.523350:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696805.523355:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696805.523361:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696805.523368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696805.523378:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134309760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696805.523385:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696805/real 1787696805] req@ffff9d85b9d2ce00 x1874531134309248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696805.523401:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696805.523407:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787696805.523408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134309248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696805.523412:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696805.523431:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4707 00000100:00080000:3.0:1787696805.523432:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696805.523440:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696805.523441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696805.523449:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134309248:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696805.523451:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696805.523460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134309376:192.168.201.137@tcp:8:kworker.0 00000008:00100000:1.0:1787696805.523461:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696805.523481:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134309888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787696805.523489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696805.523505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696805/real 1787696805] req@ffff9d85b9d2e300 x1874531134309376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696805.523519:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134309376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696805.523528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696805.523532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696805.523535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134309376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696805.523555:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134309504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696805.523575:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696805.523586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696805/real 1787696805] req@ffff9d85b9d2fb80 x1874531134309504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696805.523599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134309504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696805.523607:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696805.523611:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696805.523615:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134309504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696805.523621:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134309632:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696805.523636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696805.523645:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696805/real 1787696805] req@ffff9d85b9d2ca80 x1874531134309632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696805.523657:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134309632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696805.523666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696805.523669:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696805.523672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134309632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696805.523689:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134309760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696805.523705:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696805.523714:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696805/real 1787696805] req@ffff9d85b9d2c380 x1874531134309760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696805.523727:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134309760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696805.523735:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696805.523738:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696805.523741:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134309760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696805.523747:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134309888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696805.523762:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696805.523772:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696805/real 1787696805] req@ffff9d85b9d2f480 x1874531134309888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696805.523783:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134309888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696805.523792:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696805.523795:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696805.523798:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134309888:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696806.099097:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696807.122974:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696808.147110:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696809.170969:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696810.195083:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696810.643026:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696810.643041:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696810.643045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696810.643058:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4712 00000100:00080000:1.0:1787696810.643071:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696810.643080:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696810.643127:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134310016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696810.643176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696810.643180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696810.643182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696810.643187:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4712 00000100:00080000:1.0:1787696810.643191:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696810.643195:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696810.643205:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696810.643220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134310144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696810.643230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696810.643233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696810.643235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696810.643238:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4712 00000100:00080000:1.0:1787696810.643241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696810.643245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696810.643251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696810.643261:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134310272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696810.643269:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696810.643272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696810.643274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696810.643277:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4712 00000100:00080000:1.0:1787696810.643280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696810.643284:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696810.643289:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696810.643299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134310400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696810.643307:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696810.643311:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696810.643312:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696810.643316:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4712 00000100:00080000:1.0:1787696810.643320:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696810.643324:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696810.643328:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696810.643338:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134310528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696810.643347:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696810.643350:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696810.643351:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696810.643355:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4712 00000100:00080000:1.0:1787696810.643358:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696810.643361:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696810.643366:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696810.643375:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134310656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696810.643383:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696810.643387:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696810.643389:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696810.643392:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4712 00000100:00080000:1.0:1787696810.643396:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696810.643399:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696810.643403:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696810.643414:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134310784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696810.643562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134310016:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696810.643638:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696810.643701:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696810/real 1787696810] req@ffff9d85b9d2dc00 x1874531134310016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696810.643727:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134310016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696810.643737:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696810.643741:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696810.643748:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134310016:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696810.643756:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134310144:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696810.643783:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696810.643796:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696810/real 1787696810] req@ffff9d85b9d2f800 x1874531134310144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696810.643808:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134310144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696810.643816:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696810.643819:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696810.643822:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134310144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696810.643828:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134310272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696810.643848:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696810.643889:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696810/real 1787696810] req@ffff9d85b9d2d500 x1874531134310272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696810.643900:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134310272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696810.643908:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696810.643911:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696810.643914:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134310272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696810.643920:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134310400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696810.643940:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696810.643952:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696810/real 1787696810] req@ffff9d85b9d2ed80 x1874531134310400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696810.643963:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134310400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696810.643971:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696810.643974:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696810.643977:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134310400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696810.643983:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134310528:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696810.644003:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696810.644014:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696810/real 1787696810] req@ffff9d85b9d2c700 x1874531134310528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696810.644025:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134310528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696810.644033:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696810.644036:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696810.644039:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134310528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696810.644045:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134310656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696810.644062:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696810.644071:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696810/real 1787696810] req@ffff9d85b9d2e680 x1874531134310656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696810.644082:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134310656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696810.644090:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696810.644093:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696810.644096:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134310656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696810.644101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134310784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696810.644118:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696810.644128:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696810/real 1787696810] req@ffff9d85b9d2f480 x1874531134310784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696810.644139:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134310784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696810.644147:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696810.644150:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696810.644153:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134310784:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696811.219071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696812.243058:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696813.267009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696814.291065:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696815.315079:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696815.763137:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696815.763151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696815.763156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696815.763170:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4717 00000100:00080000:3.0:1787696815.763185:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696815.763193:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696815.763256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134310912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696815.763289:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696815.763293:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696815.763295:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696815.763301:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4717 00000100:00080000:3.0:1787696815.763306:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696815.763311:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696815.763322:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134310912:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787696815.763324:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696815.763344:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134311040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696815.763355:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696815.763359:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696815.763361:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696815.763385:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4717 00000100:00080000:3.0:1787696815.763389:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696815.763395:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696815.763401:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696815.763406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696815.763417:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134311168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696815.763429:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696815.763432:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696815.763434:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696815.763438:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4717 00000100:00080000:3.0:1787696815.763442:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696815.763445:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696815.763450:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696815.763463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696815/real 1787696815] req@ffff9d85c0fb0700 x1874531134310912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696815.763465:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134311296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696815.763487:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696815.763492:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696815.763495:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696815.763497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134310912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696815.763503:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4717 00000100:00080000:3.0:1787696815.763510:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696815.763516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696815.763517:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696815.763524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787696815.763525:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696815.763530:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134310912:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696815.763543:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134311424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696815.763554:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696815.763557:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696815.763561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311040:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696815.763562:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696815.763571:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4717 00000100:00080000:3.0:1787696815.763576:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696815.763580:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696815.763586:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696815.763594:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696815.763601:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134311552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696815.763609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696815/real 1787696815] req@ffff9d85c0fb0000 x1874531134311040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696815.763623:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134311040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696815.763631:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696815.763634:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696815.763637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311040:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696815.763642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696815.763663:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696815.763672:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696815/real 1787696815] req@ffff9d85c0fb1f80 x1874531134311168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696815.763682:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134311168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696815.763688:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696815.763692:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696815.763695:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696815.763700:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696815.763718:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696815.763728:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696815/real 1787696815] req@ffff9d85c0fb1880 x1874531134311296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696815.763739:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134311296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696815.763746:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696815.763749:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696815.763752:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696815.763765:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134311424:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696815.763785:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696815.763797:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696815/real 1787696815] req@ffff9d85c0fb3480 x1874531134311424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696815.763815:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134311424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696815.763823:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696815.763826:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696815.763829:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134311424:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696815.764029:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696815.764034:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696815.764036:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696815.764040:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4717 00000100:00080000:3.0:1787696815.764045:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696815.764049:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696815.764054:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696815.764068:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134311680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696815.765433:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134311552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696815.765468:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696815.765484:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696815/real 1787696815] req@ffff9d85c0fb3100 x1874531134311552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696815.765498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134311552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696815.765506:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696815.765510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696815.765513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134311552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696815.765519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134311680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696815.765538:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696815.765548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696815/real 1787696815] req@ffff9d85c0fb2680 x1874531134311680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696815.765557:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134311680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696815.765566:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696815.765568:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696815.765570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134311680:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696816.339111:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696817.363168:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696818.387173:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696819.411072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696820.435053:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696820.882978:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696820.882985:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696820.882988:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696820.882995:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4722 00000100:00080000:2.0:1787696820.883004:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696820.883009:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696820.883037:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134311808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696820.883056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696820.883058:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696820.883060:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696820.883062:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4722 00000100:00080000:2.0:1787696820.883064:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696820.883066:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696820.883073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696820.883088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134311936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696820.883098:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696820.883101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696820.883104:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696820.883108:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4722 00000100:00080000:2.0:1787696820.883111:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696820.883115:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696820.883120:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696820.883130:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134312064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696820.883142:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134311808:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787696820.883144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696820.883159:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696820.883160:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696820.883164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4722 00000100:00080000:2.0:1787696820.883167:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696820.883173:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696820.883179:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696820.883192:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134312192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696820.883201:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696820.883204:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696820.883207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696820.883211:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4722 00000100:00080000:2.0:1787696820.883215:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696820.883219:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696820.883224:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696820.883235:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134312320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696820.883245:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787696820.883252:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696820.883256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696820.883259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696820.883265:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4722 00000100:00080000:2.0:1787696820.883269:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696820.883274:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696820.883299:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696820.883309:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134312448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696820.883315:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696820.883317:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696820.883319:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696820.883322:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4722 00000100:00080000:2.0:1787696820.883326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696820.883332:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696820/real 1787696820] req@ffff9d85babd7800 x1874531134311808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696820.883333:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696820.883341:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696820.883355:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134312576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696820.883368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134311808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.883381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.883388:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696820.883393:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134311808:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696820.883401:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311936:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696820.883427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696820.883440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696820/real 1787696820] req@ffff9d85babd5500 x1874531134311936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.883452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134311936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.883459:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.883463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696820.883467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134311936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696820.883496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696820.883517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696820.883529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696820/real 1787696820] req@ffff9d85babd6d80 x1874531134312064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.883540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134312064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.883548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.883552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696820.883555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696820.883561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696820.883580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696820.883589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696820/real 1787696820] req@ffff9d85babd5c00 x1874531134312192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.883599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134312192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.883607:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.883610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696820.883613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696820.883620:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134312320:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696820.883636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696820.883646:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696820/real 1787696820] req@ffff9d85babd7480 x1874531134312320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.883658:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134312320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.883665:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.883668:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696820.883671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134312320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696820.883677:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134312448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696820.883699:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696820.883709:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696820/real 1787696820] req@ffff9d85babd6300 x1874531134312448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.883719:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134312448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.883726:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.883729:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696820.883732:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134312448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696820.883737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134312576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696820.883754:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696820.883763:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696820/real 1787696820] req@ffff9d85babd4380 x1874531134312576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.883775:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134312576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.883801:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.883804:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696820.883808:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134312576:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787696821.459056:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696822.483086:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696823.507024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696824.531029:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696825.555061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696826.002981:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696826.002992:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696826.002997:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696826.003009:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4727 00000100:00080000:2.0:1787696826.003022:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696826.003030:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696826.003081:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134312704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696826.003114:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696826.003117:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696826.003119:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696826.003122:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4727 00000100:00080000:2.0:1787696826.003126:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696826.003129:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696826.003138:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696826.003151:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134312832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696826.003160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696826.003163:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696826.003164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696826.003167:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4727 00000100:00080000:2.0:1787696826.003170:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696826.003174:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696826.003177:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134312704:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787696826.003179:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696826.003216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134312960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696826.003226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696826.003229:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696826.003230:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696826.003233:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4727 00000100:00080000:2.0:1787696826.003237:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696826.003240:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696826.003243:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696826.003254:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134313088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696826.003261:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696826.003263:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696826.003265:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696826.003267:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4727 00000100:00080000:2.0:1787696826.003269:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696826.003272:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787696826.003278:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787696826.003279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696826.003305:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134313216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696826.003322:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696826.003326:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696826.003329:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696826.003334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696826/real 1787696826] req@ffff9d85babd4000 x1874531134312704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696826.003338:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4727 00000100:00080000:2.0:1787696826.003347:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696826.003354:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696826.003359:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134312704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696826.003363:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696826.003373:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696826.003378:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696826.003384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134312704:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787696826.003384:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134313344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696826.003396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312832:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787696826.003408:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696826.003412:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696826.003415:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696826.003420:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4727 00000800:00000100:1.0:1787696826.003421:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787696826.003429:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696826.003437:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696826/real 1787696826] req@ffff9d85babd6680 x1874531134312832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696826.003438:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696826.003448:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696826.003453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134312832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696826.003461:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787696826.003465:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134313472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696826.003475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696826.003482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696826.003508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696826.003523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696826.003531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696826/real 1787696826] req@ffff9d85babd6a00 x1874531134312960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696826.003539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134312960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696826.003544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696826.003546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696826.003549:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134312960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696826.003552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696826.003569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696826.003576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696826/real 1787696826] req@ffff9d85babd5180 x1874531134313088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696826.003584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134313088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696826.003589:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696826.003591:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696826.003593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696826.003596:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134313216:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696826.003607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696826.003614:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696826/real 1787696826] req@ffff9d85babd4a80 x1874531134313216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696826.003622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134313216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696826.003627:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696826.003629:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696826.003631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134313216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696826.003634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134313344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696826.003652:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696826.003662:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696826/real 1787696826] req@ffff9d85babd7b80 x1874531134313344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696826.003670:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134313344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696826.003675:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696826.003678:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696826.003680:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134313344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696826.003683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134313472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696826.003694:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696826.003699:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696826/real 1787696826] req@ffff9d85babd7100 x1874531134313472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696826.003706:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134313472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696826.003712:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696826.003715:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696826.003717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134313472:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696826.579055:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696827.602964:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696828.626938:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696829.650979:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696830.674987:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696831.122942:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696831.122950:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696831.122954:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696831.122965:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4732 00000100:00080000:2.0:1787696831.122976:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696831.122983:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696831.123021:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134313600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696831.123052:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696831.123056:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696831.123057:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696831.123062:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4732 00000100:00080000:2.0:1787696831.123065:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696831.123069:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696831.123077:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696831.123090:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134313728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696831.123097:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696831.123120:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696831.123122:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134313600:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787696831.123122:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696831.123128:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4732 00000100:00080000:2.0:1787696831.123133:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696831.123135:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696831.123139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696831.123149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134313856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696831.123160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696831.123163:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696831.123164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696831.123166:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4732 00000100:00080000:2.0:1787696831.123169:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696831.123171:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696831.123173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696831.123183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134313984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696831.123185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787696831.123192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696831.123194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696831.123196:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696831.123198:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4732 00000100:00080000:2.0:1787696831.123201:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696831.123203:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696831.123206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696831.123212:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134314112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696831.123217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696831.123220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696831.123221:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696831.123224:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4732 00000100:00080000:2.0:1787696831.123227:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696831.123231:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696831.123233:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696831/real 1787696831] req@ffff9d85babd4e00 x1874531134313600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696831.123234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696831.123244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134314240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696831.123255:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696831.123257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696831.123259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696831.123261:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134313600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696831.123263:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4732 00000100:00080000:2.0:1787696831.123268:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696831.123274:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696831.123279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696831.123283:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787696831.123288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134314368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696831.123291:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696831.123299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134313600:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696831.123309:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313728:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696831.123329:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696831.123340:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696831/real 1787696831] req@ffff9d85babd5f80 x1874531134313728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696831.123351:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134313728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696831.123359:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696831.123362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696831.123366:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696831.123387:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696831.123405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696831.123414:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696831/real 1787696831] req@ffff9d85babd4700 x1874531134313856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696831.123425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134313856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696831.123432:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696831.123435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696831.123438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696831.123444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696831.123460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696831.123468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696831/real 1787696831] req@ffff9d85babd7100 x1874531134313984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696831.123479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134313984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696831.123487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696831.123490:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696831.123493:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134313984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696831.123498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134314112:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696831.123511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696831.123519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696831/real 1787696831] req@ffff9d85babd7b80 x1874531134314112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696831.123529:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134314112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696831.123537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696831.123539:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696831.123542:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134314112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696831.123548:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134314240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696831.123565:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696831.123575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696831/real 1787696831] req@ffff9d85babd4a80 x1874531134314240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696831.123586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134314240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696831.123593:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696831.123596:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696831.123599:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134314240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696831.123604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134314368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696831.123618:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696831.123626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696831/real 1787696831] req@ffff9d85babd5180 x1874531134314368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696831.123636:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134314368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696831.123644:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696831.123647:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696831.123650:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134314368:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696831.699017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696832.723034:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696833.746987:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696834.771047:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696835.795010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696836.242977:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696836.242985:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696836.242989:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696836.242997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4738 00000100:00080000:3.0:1787696836.243006:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696836.243033:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696836.243065:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134314496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696836.243087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696836.243090:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696836.243091:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696836.243094:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4738 00000100:00080000:3.0:1787696836.243097:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696836.243099:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696836.243107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696836.243115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134314624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696836.243126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696836.243128:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696836.243130:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696836.243132:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4738 00000100:00080000:3.0:1787696836.243134:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696836.243136:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696836.243140:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696836.243146:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134314496:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696836.243148:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134314752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696836.243162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696836.243164:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696836.243165:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696836.243167:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4738 00000100:00080000:3.0:1787696836.243169:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696836.243171:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696836.243174:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696836.243182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134314880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696836.243188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696836.243190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696836.243191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696836.243193:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4738 00000100:00080000:3.0:1787696836.243197:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696836.243200:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696836.243203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696836.243209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134315008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696836.243215:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696836.243217:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696836.243218:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696836.243220:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4738 00000100:00080000:3.0:1787696836.243223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696836.243227:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696836.243228:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696836.243234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696836.243240:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134315136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696836.243246:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696836.243248:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696836.243250:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696836.243252:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4738 00000100:00080000:3.0:1787696836.243255:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696836.243257:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696836.243260:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696836.243267:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134315264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696836.243293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696836/real 1787696836] req@ffff9d85c0fb1500 x1874531134314496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696836.243315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134314496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696836.243324:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696836.243326:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696836.243329:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134314496:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696836.243335:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134314624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696836.243354:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696836.243363:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696836/real 1787696836] req@ffff9d85c0fb2300 x1874531134314624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696836.243371:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134314624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696836.243376:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696836.243379:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696836.243381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134314624:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696836.243399:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134314752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696836.243410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696836.243417:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696836/real 1787696836] req@ffff9d85c0fb2a00 x1874531134314752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696836.243424:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134314752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696836.243429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696836.243431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696836.243433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134314752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696836.243436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134314880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696836.243446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696836.243451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696836/real 1787696836] req@ffff9d85c0fb0a80 x1874531134314880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696836.243458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134314880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696836.243464:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696836.243467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696836.243470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134314880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696836.243476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315008:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696836.243494:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696836.243506:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696836/real 1787696836] req@ffff9d85c0fb0e00 x1874531134315008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696836.243519:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134315008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696836.243526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696836.243528:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696836.243530:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696836.243534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696836.243548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696836.243555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696836/real 1787696836] req@ffff9d85c0fb3800 x1874531134315136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696836.243562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134315136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696836.243568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696836.243570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696836.243572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696836.243575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696836.243587:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696836.243597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696836/real 1787696836] req@ffff9d85c0fb1c00 x1874531134315264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696836.243609:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134315264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696836.243618:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696836.243621:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696836.243625:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315264:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696836.819083:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696837.843136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696838.867012:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696839.891029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696840.914978:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696841.362969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696841.362980:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696841.362984:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696841.362994:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4743 00000100:00080000:3.0:1787696841.363013:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696841.363020:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696841.363057:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134315392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696841.363085:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696841.363089:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696841.363091:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696841.363096:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4743 00000100:00080000:3.0:1787696841.363099:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696841.363103:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696841.363113:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696841.363125:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134315520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696841.363138:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134315392:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696841.363139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696841.363148:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696841.363151:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696841.363156:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4743 00000100:00080000:3.0:1787696841.363159:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696841.363163:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696841.363168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696841.363188:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134315648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696841.363199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696841.363203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696841.363205:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696841.363209:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4743 00000100:00080000:3.0:1787696841.363214:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696841.363221:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696841.363221:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696841.363230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696841.363245:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134315776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696841.363257:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696841.363261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696841.363263:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696841.363269:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696841/real 1787696841] req@ffff9d85c0fb3b80 x1874531134315392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696841.363270:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4743 00000100:00080000:3.0:1787696841.363276:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696841.363284:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696841.363292:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696841.363302:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134315392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696841.363305:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134315904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696841.363331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696841.363332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696841.363340:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696841.363341:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696841.363343:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696841.363349:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4743 00000100:00080000:3.0:1787696841.363352:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696841.363360:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134315392:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696841.363360:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696841.363375:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696841.363387:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134316032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696841.363391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134315520:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696841.363408:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696841.363412:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696841.363414:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696841.363419:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4743 00000100:00080000:3.0:1787696841.363423:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696841.363426:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696841.363431:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696841.363436:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696841.363448:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134316160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696841.363457:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696841/real 1787696841] req@ffff9d85c0fb0380 x1874531134315520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696841.363474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134315520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696841.363484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696841.363489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696841.363494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134315520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696841.363500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134315648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696841.363519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696841.363529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696841/real 1787696841] req@ffff9d85c0fb2d80 x1874531134315648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696841.363543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134315648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696841.363551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696841.363555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696841.363558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134315648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696841.363564:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134315776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696841.363580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696841.363589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696841/real 1787696841] req@ffff9d85c0fb1c00 x1874531134315776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696841.363602:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134315776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696841.363610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696841.363613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696841.363617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134315776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696841.363622:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315904:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696841.363637:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696841.363647:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696841/real 1787696841] req@ffff9d85c0fb3800 x1874531134315904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696841.363659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134315904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696841.363668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696841.363671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696841.363674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134315904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696841.363690:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696841.363705:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696841.363714:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696841/real 1787696841] req@ffff9d85c0fb0e00 x1874531134316032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696841.363725:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134316032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696841.363734:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696841.363738:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696841.363741:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696841.363747:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696841.363761:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696841.363770:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696841/real 1787696841] req@ffff9d85c0fb0a80 x1874531134316160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696841.363782:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134316160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696841.363790:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696841.363793:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696841.363797:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316160:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696841.939046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696842.963038:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696843.987005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696845.011029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696846.034956:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696846.482909:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696846.482917:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696846.482921:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696846.482929:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4748 00000100:00080000:3.0:1787696846.482946:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696846.482952:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696846.482984:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134316288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696846.483008:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696846.483012:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696846.483014:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696846.483018:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4748 00000100:00080000:3.0:1787696846.483022:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696846.483026:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696846.483035:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696846.483049:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134316416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696846.483059:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134316288:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696846.483059:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696846.483062:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696846.483064:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696846.483067:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4748 00000100:00080000:3.0:1787696846.483070:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696846.483074:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696846.483087:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696846.483097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134316544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696846.483105:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696846.483108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696846.483110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696846.483113:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4748 00000100:00080000:3.0:1787696846.483116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696846.483119:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696846.483124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696846.483134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134316672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696846.483142:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696846.483146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696846.483148:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696846.483151:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4748 00000100:00080000:3.0:1787696846.483155:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696846.483159:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696846.483164:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696846.483166:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134316416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696846.483175:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134316800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696846.483185:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696846.483188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696846.483191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696846.483195:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4748 00000100:00080000:3.0:1787696846.483198:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696846.483199:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134316544:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787696846.483202:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696846.483207:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696846.483217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134316928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696846.483223:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134316672:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787696846.483227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696846.483231:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696846.483232:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696846.483236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4748 00000100:00080000:3.0:1787696846.483239:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696846.483243:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316800:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696846.483243:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696846.483248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696846.483258:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134317056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696846.483262:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696846.483285:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317056:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696847.059067:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696848.082964:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696849.106999:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696849.555098:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696849.555135:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696849.555151:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696849.555185:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696849.555193:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696849.555199:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696849.555204:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696849.555210:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696849.555217:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787696849.555245:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696846/real 1787696849] req@ffff9d85c0fb1880 x1874531134317056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696849.555279:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134317056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696849.555297:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696849.555301:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696849.555305:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696849.555313:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696846/real 1787696849] req@ffff9d85c0fb3100 x1874531134316800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696849.555326:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134316800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696849.555334:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696849.555338:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696849.555342:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696849.555349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696846/real 1787696849] req@ffff9d85c0fb3480 x1874531134316928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696849.555362:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134316928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696849.555379:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696849.555382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696849.555385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134316928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696849.555389:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696846/real 1787696849] req@ffff9d85c0fb2300 x1874531134316416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696849.555400:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134316416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696849.555408:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696849.555412:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696849.555415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134316416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696849.555423:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696846/real 1787696849] req@ffff9d85c0fb1500 x1874531134316544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696849.555436:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134316544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696849.555445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696849.555457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696849.555461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134316544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696849.555468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696846/real 1787696849] req@ffff9d85c0fb2680 x1874531134316672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696849.555481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134316672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696849.555490:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696849.555494:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696849.555498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134316672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696849.555505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696846/real 1787696849] req@ffff9d85c0fb2a00 x1874531134316288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696849.555518:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134316288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696849.555526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696849.555529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696849.555532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134316288:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787696850.130974:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696851.155027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696851.602953:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696851.602962:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696851.602966:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696851.602984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4753 00000100:00080000:3.0:1787696851.603000:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696851.603009:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696851.603044:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134317184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696851.603076:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696851.603080:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696851.603082:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696851.603086:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4753 00000100:00080000:3.0:1787696851.603090:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696851.603094:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696851.603103:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696851.603116:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134317312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696851.603126:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696851.603129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696851.603131:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696851.603135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4753 00000100:00080000:3.0:1787696851.603139:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696851.603144:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134317184:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696851.603146:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696851.603155:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696851.603167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134317440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696851.603177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696851.603185:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696851.603188:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696851.603192:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4753 00000100:00080000:3.0:1787696851.603196:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696851.603200:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696851.603205:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696851.603217:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696851.603221:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134317568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696851.603239:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696851.603243:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696851.603245:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696851.603250:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4753 00000100:00080000:3.0:1787696851.603254:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696851.603259:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696851/real 1787696851] req@ffff9d85c0fb1f80 x1874531134317184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696851.603263:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696851.603271:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696851.603288:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134317696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696851.603293:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134317184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696851.603303:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696851.603312:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696851.603317:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696851.603319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696851.603322:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4753 00000100:00080000:1.0:1787696851.603329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696851.603329:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696851.603339:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696851.603341:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134317184:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787696851.603345:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696851.603353:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134317312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696851.603359:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134317824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696851.603370:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696851.603373:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696851.603375:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696851.603378:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4753 00000100:00080000:3.0:1787696851.603381:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696851.603385:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696851.603389:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696851.603401:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134317952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696851.603405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696851.603429:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696851/real 1787696851] req@ffff9d85c0fb0000 x1874531134317312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696851.603446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134317312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696851.603456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696851.603460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696851.603463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134317312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696851.603489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134317440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696851.603513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696851.603526:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696851/real 1787696851] req@ffff9d85c0fb0700 x1874531134317440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696851.603540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134317440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696851.603550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696851.603553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696851.603557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134317440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696851.603563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134317568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696851.603584:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696851.603619:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696851/real 1787696851] req@ffff9d85c0fb2a00 x1874531134317568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696851.603632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134317568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696851.603640:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696851.603643:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696851.603645:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134317568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696851.603651:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317696:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696851.603672:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696851.603684:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696851/real 1787696851] req@ffff9d85c0fb2680 x1874531134317696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696851.603696:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134317696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696851.603709:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696851.603712:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696851.603715:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696851.603721:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696851.603742:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696851.603774:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696851/real 1787696851] req@ffff9d85c0fb1500 x1874531134317824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696851.603787:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134317824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696851.603796:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696851.603799:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696851.603802:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696851.603816:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696851.603838:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696851.603883:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696851/real 1787696851] req@ffff9d85c0fb2300 x1874531134317952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696851.603897:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134317952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696851.603906:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696851.603910:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696851.603915:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134317952:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787696852.178985:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696853.203003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696854.227023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696855.251015:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696856.275000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696856.722935:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696856.722947:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696856.722952:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696856.722964:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4758 00000100:00080000:3.0:1787696856.722976:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696856.722984:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696856.723030:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134318080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696856.723076:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696856.723082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696856.723084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696856.723089:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4758 00000100:00080000:3.0:1787696856.723094:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696856.723099:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696856.723112:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696856.723124:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134318080:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696856.723132:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134318208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696856.723153:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696856.723157:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696856.723160:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696856.723165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4758 00000100:00080000:3.0:1787696856.723170:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696856.723175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696856.723182:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696856.723195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134318336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696856.723203:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696856.723207:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696856.723211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696856.723212:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696856.723216:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4758 00000100:00080000:3.0:1787696856.723221:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696856.723224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696856.723227:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696856.723236:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134318464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696856.723243:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696856.723245:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696856.723246:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696856.723248:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4758 00000100:00080000:3.0:1787696856.723252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696856.723255:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696856.723265:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696856/real 1787696856] req@ffff9d85c0fb3480 x1874531134318080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696856.723265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696856.723277:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134318592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696856.723288:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696856.723290:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696856.723291:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696856.723296:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4758 00000100:00100000:1.0:1787696856.723299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134318080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696856.723304:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696856.723312:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696856.723319:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696856.723325:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696856.723328:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134318720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696856.723335:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696856.723342:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696856.723343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134318080:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696856.723348:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696856.723351:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696856.723354:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4758 00000100:00080000:3.0:1787696856.723356:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696856.723359:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696856.723361:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696856.723368:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134318848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696856.723377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134318208:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696856.723412:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696856.723428:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696856/real 1787696856] req@ffff9d85c0fb3100 x1874531134318208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696856.723443:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134318208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696856.723453:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696856.723457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696856.723460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134318208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696856.723465:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134318336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696856.723485:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696856.723493:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696856/real 1787696856] req@ffff9d85c0fb1880 x1874531134318336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696856.723502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134318336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696856.723507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696856.723509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696856.723522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134318336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696856.723526:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134318464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696856.723539:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696856.723546:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696856/real 1787696856] req@ffff9d85c0fb0a80 x1874531134318464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696856.723554:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134318464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696856.723560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696856.723562:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696856.723565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134318464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696856.723568:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134318592:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696856.723579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696856.723585:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696856/real 1787696856] req@ffff9d85c0fb0e00 x1874531134318592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696856.723598:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134318592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696856.723604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696856.723605:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696856.723607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134318592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696856.723610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134318720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696856.723620:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696856.723625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696856/real 1787696856] req@ffff9d85c0fb3800 x1874531134318720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696856.723632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134318720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696856.723637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696856.723640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696856.723644:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134318720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696856.723676:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134318848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696856.723699:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696856.723712:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696856/real 1787696856] req@ffff9d85c0fb1c00 x1874531134318848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696856.723720:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134318848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696856.723731:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696856.723733:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696856.723735:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134318848:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696857.299023:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696858.323001:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696859.347009:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696860.371083:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696861.395047:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696861.842978:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696861.842988:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696861.842992:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696861.843003:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4763 00000100:00080000:2.0:1787696861.843013:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696861.843021:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696861.843064:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134318976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696861.843091:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696861.843094:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696861.843096:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696861.843100:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4763 00000100:00080000:2.0:1787696861.843104:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696861.843108:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696861.843117:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696861.843132:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134319104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696861.843147:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134318976:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787696861.843152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696861.843157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696861.843159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696861.843164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4763 00000100:00080000:2.0:1787696861.843168:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696861.843172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696861.843178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696861.843190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134319232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696861.843199:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696861.843208:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696861.843210:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696861.843215:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787696861.843217:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4763 00000100:00080000:2.0:1787696861.843222:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696861.843226:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696861.843232:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696861.843243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134319360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696861.843253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696861.843256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696861.843258:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696861.843261:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4763 00000100:00080000:2.0:1787696861.843265:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696861.843269:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696861.843274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696861.843278:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696861/real 1787696861] req@ffff9d85babd6a00 x1874531134318976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696861.843289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134319488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696861.843302:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696861.843305:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134318976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696861.843307:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696861.843312:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696861.843323:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4763 00000100:00080000:1.0:1787696861.843328:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696861.843328:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696861.843332:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696861.843338:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787696861.843339:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696861.843346:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134318976:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787696861.843352:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134319616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696861.843360:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696861.843363:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696861.843364:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696861.843366:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4763 00000100:00080000:2.0:1787696861.843370:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696861.843373:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696861.843376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134319104:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787696861.843378:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696861.843390:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134319744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696861.843410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696861.843425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696861/real 1787696861] req@ffff9d85babd6680 x1874531134319104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696861.843437:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134319104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696861.843446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696861.843449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696861.843454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134319104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696861.843460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134319232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696861.843476:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696861.843486:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696861/real 1787696861] req@ffff9d85babd4000 x1874531134319232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696861.843497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134319232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696861.843505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696861.843508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696861.843512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134319232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696861.843518:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134319360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696861.843533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696861.843542:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696861/real 1787696861] req@ffff9d85babd4380 x1874531134319360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696861.843554:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134319360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696861.843562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696861.843565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696861.843568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134319360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696861.843604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134319488:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696861.843623:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696861.843634:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696861/real 1787696861] req@ffff9d85babd6300 x1874531134319488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696861.843646:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134319488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696861.843654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696861.843657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696861.843661:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134319488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696861.843666:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134319616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696861.843682:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696861.843692:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696861/real 1787696861] req@ffff9d85babd7480 x1874531134319616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696861.843703:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134319616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696861.843711:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696861.843714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696861.843717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134319616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696861.843734:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134319744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696861.843750:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696861.843759:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696861/real 1787696861] req@ffff9d85babd5c00 x1874531134319744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696861.843771:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134319744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696861.843779:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696861.843782:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696861.843786:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134319744:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696862.419015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696863.443050:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696864.467026:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696865.491046:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696866.515013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696866.962950:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696866.962960:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696866.962964:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696866.962987:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4768 00000100:00080000:2.0:1787696866.962998:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696866.963006:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696866.963039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134319872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696866.963061:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696866.963064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696866.963065:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696866.963068:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4768 00000100:00080000:2.0:1787696866.963071:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696866.963075:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696866.963082:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696866.963093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134320000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696866.963102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696866.963105:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134319872:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787696866.963105:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696866.963108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696866.963111:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4768 00000100:00080000:2.0:1787696866.963113:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696866.963116:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696866.963119:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696866.963128:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134320128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696866.963135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696866.963137:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696866.963138:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696866.963142:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4768 00000100:00080000:2.0:1787696866.963146:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696866.963150:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696866.963154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696866.963168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134320256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696866.963178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696866.963182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787696866.963185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787696866.963186:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696866.963194:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4768 00000100:00080000:2.0:1787696866.963199:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696866.963203:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696866.963208:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696866.963218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134320384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696866.963226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696866.963230:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696866/real 1787696866] req@ffff9d85babd6d80 x1874531134319872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696866.963231:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696866.963235:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696866.963244:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4768 00000100:00080000:2.0:1787696866.963248:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696866.963252:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696866.963255:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134319872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696866.963259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696866.963266:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696866.963271:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696866.963275:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134319872:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787696866.963280:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134320512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696866.963292:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696866.963295:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696866.963297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320000:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787696866.963299:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696866.963307:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4768 00000100:00080000:2.0:1787696866.963313:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696866.963318:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787696866.963322:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787696866.963326:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696866.963335:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696866/real 1787696866] req@ffff9d85babd5500 x1874531134320000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696866.963341:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134320640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696866.963360:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134320000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696866.963367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696866.963370:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696866.963373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696866.963377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696866.963391:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696866.963397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696866/real 1787696866] req@ffff9d85babd7800 x1874531134320128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696866.963405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134320128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696866.963411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696866.963412:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696866.963415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696866.963418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696866.963428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696866.963434:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696866/real 1787696866] req@ffff9d85babd5c00 x1874531134320256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696866.963441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134320256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696866.963446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696866.963447:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696866.963449:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696866.963453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134320384:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696866.963462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696866.963468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696866/real 1787696866] req@ffff9d85babd7480 x1874531134320384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696866.963475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134320384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696866.963502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696866.963505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696866.963507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134320384:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696866.963511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134320512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696866.963526:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696866.963533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696866/real 1787696866] req@ffff9d85babd6300 x1874531134320512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696866.963541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134320512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696866.963546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696866.963548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696866.963550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134320512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696866.963562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134320640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696866.963572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696866.963578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696866/real 1787696866] req@ffff9d85babd4380 x1874531134320640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696866.963589:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134320640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696866.963597:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696866.963600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696866.963603:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134320640:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696867.539080:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696868.563014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696869.587004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696870.610984:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696871.635010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696872.083004:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696872.083015:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696872.083019:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696872.083030:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4773 00000100:00080000:3.0:1787696872.083042:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696872.083050:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696872.083089:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134320768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696872.083118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696872.083129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696872.083132:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696872.083136:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4773 00000100:00080000:3.0:1787696872.083141:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696872.083146:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696872.083156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696872.083174:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134320768:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696872.083175:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134320896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696872.083193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696872.083198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696872.083201:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696872.083205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4773 00000100:00080000:3.0:1787696872.083209:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696872.083214:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696872.083220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696872.083231:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134321024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696872.083240:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696872.083244:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696872.083246:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696872.083251:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696872.083256:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4773 00000100:00080000:3.0:1787696872.083262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696872.083267:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696872.083274:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696872.083290:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134321152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696872.083299:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696872.083302:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696872.083303:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696872.083309:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4773 00000100:00100000:1.0:1787696872.083310:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696872/real 1787696872] req@ffff9d85c0fb2d80 x1874531134320768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696872.083313:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696872.083320:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696872.083329:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696872.083346:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134320768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696872.083346:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134321280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696872.083367:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696872.083372:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696872.083373:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696872.083375:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696872.083377:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4773 00000100:00080000:3.0:1787696872.083384:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696872.083387:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696872.083390:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696872.083418:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696872.083427:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134321408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696872.083432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134320768:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696872.083437:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696872.083442:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696872.083444:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696872.083454:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4773 00000100:00080000:3.0:1787696872.083457:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696872.083460:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696872.083464:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696872.083468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696872.083474:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134321536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696872.083503:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696872.083519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696872/real 1787696872] req@ffff9d85c0fb0380 x1874531134320896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696872.083532:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134320896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696872.083541:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696872.083544:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696872.083548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134320896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696872.083554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696872.083574:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696872.083586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696872/real 1787696872] req@ffff9d85c0fb3b80 x1874531134321024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696872.083599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134321024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696872.083606:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696872.083610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696872.083613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696872.083620:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696872.083641:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696872.083653:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696872/real 1787696872] req@ffff9d85c0fb1c00 x1874531134321152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696872.083665:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134321152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696872.083674:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696872.083677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696872.083681:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696872.083687:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134321280:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696872.083705:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696872.083717:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696872/real 1787696872] req@ffff9d85c0fb3800 x1874531134321280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696872.083729:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134321280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696872.083737:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696872.083740:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696872.083753:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134321280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696872.083760:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134321408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696872.083781:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696872.083792:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696872/real 1787696872] req@ffff9d85c0fb0e00 x1874531134321408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696872.083806:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134321408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696872.083814:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696872.083818:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696872.083821:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134321408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696872.083839:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134321536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696872.083890:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696872.083902:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696872/real 1787696872] req@ffff9d85c0fb0a80 x1874531134321536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696872.083914:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134321536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696872.083922:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696872.083925:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696872.083937:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134321536:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696872.659094:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696873.682998:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696874.707039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696875.730993:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696876.754975:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696877.202995:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696877.203006:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696877.203011:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696877.203022:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4778 00000100:00080000:1.0:1787696877.203034:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696877.203042:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696877.203080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134321664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696877.203109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696877.203113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696877.203116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696877.203121:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4778 00000100:00080000:1.0:1787696877.203126:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696877.203131:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696877.203137:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134321664:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787696877.203144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696877.203162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134321792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696877.203175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696877.203180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696877.203182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696877.203186:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4778 00000100:00080000:1.0:1787696877.203191:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696877.203195:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696877.203200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696877.203203:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696877.203215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134321920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696877.203228:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696877.203232:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696877.203234:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696877.203239:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4778 00000100:00080000:1.0:1787696877.203244:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696877.203249:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696877.203254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696877.203257:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696877/real 1787696877] req@ffff9d85b9d2c380 x1874531134321664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696877.203270:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134322048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696877.203280:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134321664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696877.203284:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696877.203294:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696877.203299:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696877.203302:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696877.203323:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4778 00000100:00080000:3.0:1787696877.203329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787696877.203330:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696877.203337:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134321664:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696877.203339:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696877.203349:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696877.203363:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134322176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696877.203372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321792:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696877.203378:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696877.203382:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696877.203385:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696877.203391:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4778 00000100:00080000:1.0:1787696877.203395:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696877.203400:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696877.203407:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696877.203416:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696877.203421:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134322304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696877.203432:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787696877.203436:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696877/real 1787696877] req@ffff9d85b9d2ca80 x1874531134321792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696877.203438:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696877.203442:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696877.203450:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4778 00000100:00080000:1.0:1787696877.203460:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696877.203465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134321792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696877.203466:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696877.203476:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787696877.203481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696877.203488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696877.203494:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134322432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696877.203495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696877.203509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696877.203537:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696877.203551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696877/real 1787696877] req@ffff9d85b9d2fb80 x1874531134321920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696877.203564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134321920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696877.203579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696877.203583:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696877.203586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134321920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696877.203592:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696877.203613:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696877.203625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696877/real 1787696877] req@ffff9d85b9d2e300 x1874531134322048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696877.203637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134322048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696877.203645:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696877.203649:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696877.203653:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696877.203668:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134322176:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696877.203689:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696877.203701:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696877/real 1787696877] req@ffff9d85b9d2ce00 x1874531134322176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696877.203713:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134322176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696877.203721:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696877.203724:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696877.203728:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134322176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696877.203734:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134322304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696877.203754:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696877.203766:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696877/real 1787696877] req@ffff9d85b9d2c000 x1874531134322304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696877.203778:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134322304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696877.203786:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696877.203789:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696877.203793:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134322304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696877.203799:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134322432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696877.203818:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696877.203829:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696877/real 1787696877] req@ffff9d85b9d2c380 x1874531134322432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696877.203841:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134322432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696877.203872:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696877.203877:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696877.203880:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134322432:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696877.779012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696878.803010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696879.826972:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696880.851048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696881.875064:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696882.322986:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696882.322995:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696882.322999:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696882.323009:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4784 00000100:00080000:3.0:1787696882.323025:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696882.323031:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696882.323075:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134322560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696882.323108:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696882.323112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696882.323114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696882.323118:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4784 00000100:00080000:3.0:1787696882.323122:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696882.323126:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696882.323134:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696882.323146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134322688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696882.323154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696882.323163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696882.323164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696882.323167:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4784 00000100:00080000:3.0:1787696882.323170:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696882.323173:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134322560:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696882.323175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696882.323183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696882.323190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134322816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696882.323196:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696882.323198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696882.323205:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696882.323211:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4784 00000100:00080000:3.0:1787696882.323242:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696882.323247:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696882.323252:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696882.323264:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134322944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696882.323266:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696882.323276:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696882.323281:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696882.323283:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696882.323286:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4784 00000100:00080000:3.0:1787696882.323304:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696882.323312:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696882.323317:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696882.323328:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134323072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696882.323334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696882/real 1787696882] req@ffff9d85c0fb1880 x1874531134322560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696882.323336:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696882.323343:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696882.323346:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696882.323353:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4784 00000100:00080000:3.0:1787696882.323356:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696882.323359:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696882.323362:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696882.323366:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134322560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696882.323371:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134323200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696882.323383:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696882.323390:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696882.323392:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696882.323398:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4784 00000100:00080000:1.0:1787696882.323401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696882.323402:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696882.323408:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696882.323411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787696882.323415:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696882.323418:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134322560:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696882.323426:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134323328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696882.323429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322688:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696882.323456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696882.323469:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696882/real 1787696882] req@ffff9d85c0fb3100 x1874531134322688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696882.323481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134322688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696882.323489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696882.323492:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696882.323496:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696882.323513:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696882.323528:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696882.323535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696882/real 1787696882] req@ffff9d85c0fb3480 x1874531134322816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696882.323542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134322816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696882.323551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696882.323553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696882.323555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696882.323558:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696882.323566:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696882.323572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696882/real 1787696882] req@ffff9d85c0fb2300 x1874531134322944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696882.323578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134322944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696882.323583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696882.323586:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696882.323589:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134322944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696882.323595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323072:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696882.323611:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696882.323621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696882/real 1787696882] req@ffff9d85c0fb1500 x1874531134323072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696882.323632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134323072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696882.323653:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696882.323657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696882.323661:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696882.323666:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696882.323685:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696882.323696:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696882/real 1787696882] req@ffff9d85c0fb2680 x1874531134323200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696882.323707:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134323200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696882.323713:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696882.323716:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696882.323719:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696882.323724:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696882.323740:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696882.323751:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696882/real 1787696882] req@ffff9d85c0fb2a00 x1874531134323328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696882.323761:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134323328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696882.323768:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696882.323771:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696882.323773:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323328:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787696882.899008:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696883.923054:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696884.946980:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696885.971048:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696886.995042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696887.443009:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696887.443024:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696887.443029:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696887.443041:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4789 00000100:00080000:3.0:1787696887.443054:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696887.443063:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696887.443114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134323456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696887.443178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696887.443187:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696887.443189:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696887.443195:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4789 00000100:00080000:3.0:1787696887.443199:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696887.443204:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696887.443216:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696887.443231:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134323456:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696887.443233:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134323584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696887.443248:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696887.443252:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696887.443253:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696887.443258:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4789 00000100:00080000:3.0:1787696887.443262:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696887.443266:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696887.443272:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696887.443284:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134323712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696887.443294:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696887.443297:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696887.443299:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696887.443303:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4789 00000800:00000100:1.0:1787696887.443312:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696887.443315:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696887.443321:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696887.443327:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696887.443343:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134323840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696887.443353:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696887.443357:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696887/real 1787696887] req@ffff9d85c0fb0700 x1874531134323456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696887.443359:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696887.443363:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696887.443372:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4789 00000100:00080000:3.0:1787696887.443377:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696887.443379:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134323456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696887.443384:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696887.443392:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787696887.443392:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696887.443398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696887.443402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134323456:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696887.443407:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134323968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696887.443417:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696887.443421:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696887.443423:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696887.443425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134323584:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696887.443429:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4789 00000100:00080000:3.0:1787696887.443433:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696887.443439:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696887.443445:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696887.443456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696887.443460:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134324096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696887.443468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696887/real 1787696887] req@ffff9d85c0fb0000 x1874531134323584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.443496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134323584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.443502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.443510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696887.443513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134323584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696887.443517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134323712:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787696887.443526:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696887.443531:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787696887.443532:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696887.443536:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696887.443543:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4789 00000100:00100000:1.0:1787696887.443544:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696887/real 1787696887] req@ffff9d85c0fb1f80 x1874531134323712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696887.443549:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696887.443558:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696887.443565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134323712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696887.443566:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696887.443576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.443579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696887.443582:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134323712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696887.443586:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134324224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696887.443589:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134323840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696887.443608:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696887.443615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696887/real 1787696887] req@ffff9d85c0fb2a00 x1874531134323840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.443623:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134323840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.443628:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.443630:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696887.443632:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134323840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696887.443635:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323968:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696887.443645:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696887.443652:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696887/real 1787696887] req@ffff9d85c0fb2680 x1874531134323968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.443659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134323968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.443668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.443670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696887.443671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134323968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696887.443680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696887.443689:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696887.443693:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696887/real 1787696887] req@ffff9d85c0fb1500 x1874531134324096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.443699:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134324096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.443704:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.443706:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696887.443708:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696887.443711:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696887.443719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696887.443726:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696887/real 1787696887] req@ffff9d85c0fb2300 x1874531134324224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.443736:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134324224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.443744:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.443747:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696887.443751:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324224:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696888.019065:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696889.042968:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696890.067091:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696891.091100:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696892.114989:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696892.562954:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696892.562961:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696892.562964:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696892.562972:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4794 00000100:00080000:3.0:1787696892.562980:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696892.562986:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696892.563015:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134324352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696892.563063:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696892.563066:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696892.563067:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696892.563071:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4794 00000100:00080000:3.0:1787696892.563074:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696892.563077:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696892.563084:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696892.563100:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134324480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696892.563107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696892.563109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696892.563110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696892.563113:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4794 00000100:00080000:3.0:1787696892.563115:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696892.563118:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696892.563121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696892.563127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134324608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696892.563137:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134324352:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696892.563138:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696892.563142:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696892.563143:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696892.563146:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4794 00000100:00080000:3.0:1787696892.563156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696892.563160:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696892.563162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696892.563173:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134324736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696892.563183:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696892.563186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696892.563187:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696892.563191:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4794 00000100:00080000:3.0:1787696892.563194:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696892.563198:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696892.563203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696892.563215:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134324864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696892.563233:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696892.563237:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696892.563240:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696892.563242:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696892.563246:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4794 00000100:00080000:3.0:1787696892.563251:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696892.563255:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696892.563259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696892.563273:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134324992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696892.563283:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696892.563286:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696892.563289:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696892.563293:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4794 00000100:00080000:3.0:1787696892.563297:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696892.563298:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696892/real 1787696892] req@ffff9d85c0fb3480 x1874531134324352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696892.563310:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696892.563319:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696892.563334:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134324352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696892.563336:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134325120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696892.563398:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696892.563404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696892.563408:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134324352:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696892.563417:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134324480:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696892.563445:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696892.563459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696892/real 1787696892] req@ffff9d85c0fb3100 x1874531134324480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696892.563471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134324480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696892.563477:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696892.563479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696892.563481:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134324480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696892.563509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134324608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696892.563543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696892.563552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696892/real 1787696892] req@ffff9d85c0fb1880 x1874531134324608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696892.563564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134324608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696892.563572:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696892.563576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696892.563579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134324608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696892.563585:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134324736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696892.563605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696892.563616:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696892/real 1787696892] req@ffff9d85c0fb0a80 x1874531134324736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696892.563640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134324736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696892.563652:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696892.563656:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696892.563659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134324736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696892.563666:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324864:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696892.563686:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696892.563698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696892/real 1787696892] req@ffff9d85c0fb0e00 x1874531134324864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696892.563711:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134324864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696892.563716:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696892.563718:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696892.563720:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696892.563724:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696892.563737:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696892.563743:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696892/real 1787696892] req@ffff9d85c0fb3800 x1874531134324992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696892.563750:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134324992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696892.563755:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696892.563759:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696892.563762:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134324992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696892.563767:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134325120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696892.563785:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696892.563795:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696892/real 1787696892] req@ffff9d85c0fb1c00 x1874531134325120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696892.563806:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134325120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696892.563819:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696892.563822:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696892.563825:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134325120:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696893.139042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696894.162999:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696895.186962:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696896.210996:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696897.234947:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696897.682904:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696897.682911:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696897.682914:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696897.682921:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4799 00000100:00080000:3.0:1787696897.682929:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696897.682935:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696897.682983:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134325248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696897.683004:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696897.683007:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696897.683009:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696897.683013:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4799 00000100:00080000:3.0:1787696897.683016:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696897.683020:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696897.683029:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696897.683044:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134325248:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696897.683045:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134325376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696897.683062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696897.683064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696897.683066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696897.683069:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4799 00000100:00080000:3.0:1787696897.683071:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696897.683074:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696897.683078:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696897.683088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134325504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696897.683097:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696897.683100:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696897.683102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696897.683106:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4799 00000800:00000100:1.0:1787696897.683112:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696897.683112:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696897.683119:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696897.683125:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696897.683137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134325632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696897.683143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696897.683145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696897.683146:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696897.683148:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4799 00000100:00080000:3.0:1787696897.683151:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696897.683154:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696897.683158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696897.683162:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696897/real 1787696897] req@ffff9d85c0fb3b80 x1874531134325248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696897.683168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134325760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696897.683176:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696897.683178:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696897.683182:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134325248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696897.683182:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696897.683189:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4799 00000100:00080000:3.0:1787696897.683193:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696897.683198:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696897.683201:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787696897.683201:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696897.683208:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696897.683211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134325888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696897.683215:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134325248:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696897.683226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696897.683228:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696897.683230:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696897.683232:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4799 00000100:00080000:3.0:1787696897.683234:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696897.683236:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696897.683239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696897.683240:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134325376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696897.683249:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134326016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696897.683262:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696897.683273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696897/real 1787696897] req@ffff9d85c0fb0380 x1874531134325376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696897.683281:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134325376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696897.683286:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696897.683288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696897.683291:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134325376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696897.683294:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134325504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696897.683304:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696897.683309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696897/real 1787696897] req@ffff9d85c0fb2d80 x1874531134325504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696897.683315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134325504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696897.683320:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696897.683321:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696897.683323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134325504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696897.683326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134325632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696897.683335:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696897.683339:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696897/real 1787696897] req@ffff9d85c0fb1c00 x1874531134325632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696897.683346:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134325632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696897.683350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696897.683352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696897.683353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134325632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696897.683356:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134325760:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696897.683364:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696897.683370:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696897/real 1787696897] req@ffff9d85c0fb3800 x1874531134325760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696897.683376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134325760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696897.683380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696897.683381:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696897.683383:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134325760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696897.683386:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134325888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696897.683393:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696897.683398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696897/real 1787696897] req@ffff9d85c0fb0e00 x1874531134325888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696897.683403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134325888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696897.683408:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696897.683409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696897.683411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134325888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696897.683420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696897.683429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696897.683434:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696897/real 1787696897] req@ffff9d85c0fb0a80 x1874531134326016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696897.683440:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134326016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696897.683445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696897.683447:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696897.683448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326016:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696898.259036:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696899.282968:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696900.307019:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696901.330983:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696902.355001:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696902.802963:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696902.802972:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696902.802976:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696902.802986:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4804 00000100:00080000:1.0:1787696902.802996:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696902.803003:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696902.803035:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134326144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696902.803061:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696902.803065:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696902.803066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696902.803071:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4804 00000100:00080000:1.0:1787696902.803074:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696902.803078:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696902.803087:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696902.803087:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134326144:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696902.803104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134326272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696902.803116:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696902.803119:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696902.803121:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696902.803125:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4804 00000100:00080000:1.0:1787696902.803129:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696902.803133:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696902.803139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696902.803142:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696902.803154:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134326400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696902.803163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696902.803166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696902.803168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696902.803171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4804 00000100:00080000:1.0:1787696902.803175:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696902.803178:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696902.803183:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696902/real 1787696902] req@ffff9d85b9d2f480 x1874531134326144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787696902.803189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696902.803197:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134326144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696902.803203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134326528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696902.803207:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696902.803214:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787696902.803220:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787696902.803220:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134326144:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696902.803228:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696902.803231:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696902.803236:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4804 00000100:00080000:1.0:1787696902.803239:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696902.803243:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696902.803245:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134326272:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787696902.803250:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696902.803263:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134326656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787696902.803266:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696902.803277:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696902/real 1787696902] req@ffff9d85b9d2e680 x1874531134326272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696902.803278:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696902.803286:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696902.803290:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696902.803295:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134326272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696902.803299:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4804 00000100:00080000:3.0:1787696902.803306:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696902.803308:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696902.803311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134326272:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696902.803314:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696902.803316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134326400:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787696902.803321:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696902.803328:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696902.803334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696902.803346:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696902/real 1787696902] req@ffff9d85b9d2c700 x1874531134326400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696902.803347:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134326784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696902.803364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134326400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696902.803369:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696902.803372:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696902.803375:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696902.803377:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134326400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696902.803381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134326528:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787696902.803383:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696902.803386:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696902.803391:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4804 00000800:00000100:3.0:1787696902.803395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696902.803398:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696902.803403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696902/real 1787696902] req@ffff9d85b9d2ed80 x1874531134326528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696902.803406:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696902.803415:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696902.803417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134326528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696902.803425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696902.803426:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696902.803428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134326528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696902.803433:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134326912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696902.803439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326656:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696902.803456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696902.803464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696902/real 1787696902] req@ffff9d85b9d2d500 x1874531134326656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696902.803474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134326656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696902.803479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696902.803480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696902.803482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696902.803485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696902.803493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696902.803497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696902/real 1787696902] req@ffff9d85b9d2f800 x1874531134326784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696902.803504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134326784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696902.803508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696902.803509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696902.803510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696902.803518:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696902.803526:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696902.803531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696902/real 1787696902] req@ffff9d85b9d2dc00 x1874531134326912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696902.803536:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134326912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696902.803544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696902.803546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696902.803547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134326912:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696903.379012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696904.402974:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696905.427094:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696906.451032:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696907.475010:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696907.923050:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696907.923064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696907.923070:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696907.923084:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4809 00000100:00080000:1.0:1787696907.923100:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696907.923111:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696907.923169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134327040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696907.923200:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696907.923204:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696907.923207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696907.923214:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4809 00000100:00080000:1.0:1787696907.923219:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696907.923224:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696907.923240:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134327040:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787696907.923245:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696907.923271:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134327168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696907.923285:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696907.923289:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696907.923292:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696907.923296:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4809 00000100:00080000:1.0:1787696907.923302:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696907.923307:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696907.923316:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787696907.923326:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696907.923346:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134327296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696907.923361:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696907.923366:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696907.923374:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696907.923375:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696907/real 1787696907] req@ffff9d85b9d2dc00 x1874531134327040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696907.923391:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4809 00000100:00080000:1.0:1787696907.923403:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696907.923409:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134327040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696907.923411:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696907.923432:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787696907.923435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696907.923441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696907.923446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134327040:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696907.923454:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134327424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696907.923468:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787696907.923473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134327168:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696907.923474:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696907.923483:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696907.923490:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4809 00000100:00080000:1.0:1787696907.923496:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787696907.923502:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696907.923504:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696907.923511:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696907.923517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696907/real 1787696907] req@ffff9d85b9d2f800 x1874531134327168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696907.923529:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134327552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696907.923530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134327168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696907.923592:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696907.923599:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696907.923600:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696907.923602:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696907.923618:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4809 00000100:00080000:3.0:1787696907.923619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787696907.923627:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696907.923634:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134327168:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696907.923638:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696907.923651:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696907.923655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134327296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696907.923670:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134327680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696907.923683:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787696907.923691:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787696907.923694:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696907.923699:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696907.923705:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4809 00000100:00080000:1.0:1787696907.923712:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696907.923717:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696907/real 1787696907] req@ffff9d85b9d2d500 x1874531134327296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696907.923722:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696907.923734:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696907.923747:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134327296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696907.923752:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134327808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696907.923821:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696907.923829:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696907.923836:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134327296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696907.923842:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134327424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696907.923923:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696907.923940:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696907/real 1787696907] req@ffff9d85b9d2ed80 x1874531134327424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696907.923955:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134327424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696907.923964:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696907.923968:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696907.923971:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134327424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696907.923992:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134327552:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696907.924016:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696907.924029:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696907/real 1787696907] req@ffff9d85b9d2c700 x1874531134327552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696907.924042:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134327552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696907.924051:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696907.924053:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696907.924056:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134327552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696907.924060:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134327680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696907.924076:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696907.924084:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696907/real 1787696907] req@ffff9d85b9d2e680 x1874531134327680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696907.924091:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134327680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696907.924096:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696907.924098:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696907.924100:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134327680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696907.924104:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134327808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696907.924119:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696907.924129:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696907/real 1787696907] req@ffff9d85b9d2f480 x1874531134327808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696907.924141:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134327808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696907.924151:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696907.924155:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696907.924158:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134327808:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696908.499087:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696909.523050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696910.547044:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696911.571091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696912.595048:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696913.042986:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696913.042995:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696913.043000:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696913.043012:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4814 00000100:00080000:3.0:1787696913.043023:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696913.043031:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696913.043069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134327936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696913.043097:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696913.043102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696913.043103:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696913.043118:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4814 00000100:00080000:3.0:1787696913.043123:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696913.043128:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696913.043139:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134327936:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787696913.043139:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696913.043159:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134328064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696913.043172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696913.043176:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696913.043177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696913.043181:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4814 00000100:00080000:3.0:1787696913.043186:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696913.043189:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696913.043193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696913.043211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134328192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696913.043219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696913.043222:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696913.043223:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696913.043226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4814 00000100:00080000:3.0:1787696913.043229:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696913.043232:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696913.043235:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696913.043245:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134328320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696913.043254:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696913.043259:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696913.043261:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696913.043268:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328064:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696913.043268:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4814 00000100:00080000:3.0:1787696913.043284:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696913.043292:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696913.043300:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696913.043314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696913.043318:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134328448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696913.043331:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696913.043335:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696913.043337:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696913.043341:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4814 00000100:00080000:3.0:1787696913.043344:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696913.043348:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696913.043353:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696913.043355:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696913.043370:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134328576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696913.043383:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696913.043389:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696913.043390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134328448:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696913.043392:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696913.043400:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4814 00000100:00080000:3.0:1787696913.043406:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696913.043409:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696913.043414:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696913.043425:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134328704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696913.043430:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134328576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696913.043473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134328704:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787696913.619052:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696914.643022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696915.667015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696916.115117:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696916.115158:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696916.115171:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696916.115202:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696916.115208:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696916.115215:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696916.115223:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696916.115231:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696916.115243:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787696916.115254:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696913/real 1787696916] req@ffff9d85c0fb1500 x1874531134328448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696916.115283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134328448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696916.115307:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696916.115312:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696916.115317:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134328448:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696916.115326:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696913/real 1787696916] req@ffff9d85c0fb2680 x1874531134328576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696916.115340:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134328576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696916.115347:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696916.115351:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696916.115359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134328576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696916.115378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696913/real 1787696916] req@ffff9d85c0fb3100 x1874531134328064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696916.115392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134328064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696916.115402:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696916.115407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696916.115411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696916.115418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696913/real 1787696916] req@ffff9d85c0fb3480 x1874531134328192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696916.115440:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134328192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696916.115450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696916.115453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696916.115457:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696916.115464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696913/real 1787696916] req@ffff9d85c0fb2300 x1874531134328320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696916.115476:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134328320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696916.115484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696916.115487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696916.115490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696916.115518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696913/real 1787696916] req@ffff9d85c0fb1880 x1874531134327936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696916.115531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134327936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696916.115547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696916.115551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696916.115555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134327936:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696916.115584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696913/real 1787696916] req@ffff9d85c0fb2a00 x1874531134328704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696916.115604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134328704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696916.115611:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696916.115615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696916.115619:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134328704:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696916.691015:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696917.715090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696918.162945:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696918.162953:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696918.162956:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696918.162965:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4819 00000100:00080000:3.0:1787696918.162974:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696918.162980:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696918.163015:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134328832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696918.163041:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696918.163044:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696918.163045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696918.163048:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4819 00000100:00080000:3.0:1787696918.163051:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696918.163053:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696918.163061:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696918.163076:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134328960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696918.163088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696918.163092:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696918.163096:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696918.163098:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134328832:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696918.163103:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4819 00000100:00080000:3.0:1787696918.163109:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696918.163116:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696918.163123:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696918.163136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134329088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696918.163147:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696918.163151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696918.163152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696918.163156:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4819 00000100:00080000:3.0:1787696918.163161:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696918.163165:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787696918.163171:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787696918.163174:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696918.163193:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134329216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696918.163203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696918.163207:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696918.163209:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696918.163213:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4819 00000100:00080000:3.0:1787696918.163217:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696918.163225:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696918/real 1787696918] req@ffff9d85c0fb1f80 x1874531134328832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696918.163225:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696918.163231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696918.163252:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134329344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696918.163257:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134328832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696918.163274:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696918.163280:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696918.163280:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696918.163286:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696918.163290:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696918.163296:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4819 00000100:00100000:1.0:1787696918.163300:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134328832:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696918.163304:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696918.163312:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696918.163318:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696918.163332:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696918.163335:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134329472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696918.163351:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696918.163355:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696918.163358:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696918.163362:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4819 00000800:00000100:1.0:1787696918.163368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696918.163370:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696918.163377:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696918.163384:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696918.163390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696918/real 1787696918] req@ffff9d85c0fb0000 x1874531134328960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696918.163401:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134329600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696918.163434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134328960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696918.163445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696918.163450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696918.163454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134328960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696918.163463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696918.163487:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696918.163502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696918/real 1787696918] req@ffff9d85c0fb0700 x1874531134329088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696918.163516:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134329088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696918.163525:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696918.163529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696918.163532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696918.163539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696918.163560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696918.163573:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696918/real 1787696918] req@ffff9d85c0fb2a00 x1874531134329216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696918.163585:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134329216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696918.163616:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696918.163620:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696918.163623:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696918.163629:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134329344:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696918.163653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696918.163665:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696918/real 1787696918] req@ffff9d85c0fb1880 x1874531134329344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696918.163677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134329344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696918.163686:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696918.163689:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696918.163693:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134329344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696918.163711:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134329472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696918.163733:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696918.163745:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696918/real 1787696918] req@ffff9d85c0fb2300 x1874531134329472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696918.163757:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134329472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696918.163765:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696918.163768:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696918.163771:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134329472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696918.163777:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134329600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696918.163796:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696918.163808:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696918/real 1787696918] req@ffff9d85c0fb3480 x1874531134329600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696918.163819:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134329600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696918.163827:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696918.163830:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696918.163833:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134329600:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696918.738996:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696919.762976:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696920.787048:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696921.810997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696922.835088:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696923.283016:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696923.283032:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696923.283037:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696923.283052:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4825 00000100:00080000:3.0:1787696923.283069:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696923.283079:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696923.283135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134329728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696923.283176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696923.283180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696923.283182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696923.283188:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4825 00000100:00080000:3.0:1787696923.283192:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696923.283196:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696923.283208:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696923.283223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134329856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696923.283235:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696923.283241:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696923.283242:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134329728:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696923.283244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696923.283253:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4825 00000100:00080000:3.0:1787696923.283257:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696923.283260:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696923.283267:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696923.283280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134329984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696923.283290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696923.283294:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696923.283296:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696923.283300:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4825 00000100:00080000:3.0:1787696923.283307:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696923.283311:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696923.283316:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696923.283329:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134330112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696923.283340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696923.283343:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696923.283349:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696923.283351:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696923.283357:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4825 00000100:00080000:3.0:1787696923.283362:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696923.283367:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696923.283373:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696923.283385:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134330240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696923.283396:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696923.283400:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696923.283402:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696923.283407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696923/real 1787696923] req@ffff9d85c0fb3100 x1874531134329728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696923.283409:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4825 00000100:00080000:3.0:1787696923.283416:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696923.283424:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696923.283435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134329728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696923.283442:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696923.283446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696923.283451:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696923.283455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134329728:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696923.283460:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134330368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696923.283480:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696923.283485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329856:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696923.283485:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696923.283489:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696923.283500:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4825 00000100:00080000:3.0:1787696923.283504:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696923.283522:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696923.283529:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696923.283535:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696923.283546:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134330496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696923.283551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696923/real 1787696923] req@ffff9d85c0fb2680 x1874531134329856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696923.283579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134329856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696923.283586:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696923.283588:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696923.283591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696923.283596:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696923.283621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696923.283628:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696923/real 1787696923] req@ffff9d85c0fb1500 x1874531134329984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696923.283636:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134329984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696923.283642:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696923.283645:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696923.283647:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134329984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696923.283650:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134330112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696923.283661:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696923.283667:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696923/real 1787696923] req@ffff9d85c0fb0a80 x1874531134330112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696923.283680:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134330112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696923.283685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696923.283687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696923.283689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134330112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696923.283692:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134330240:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696923.283703:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696923.283709:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696923/real 1787696923] req@ffff9d85c0fb0e00 x1874531134330240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696923.283717:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134330240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696923.283722:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696923.283724:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696923.283726:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134330240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696923.283729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134330368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696923.283738:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696923.283744:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696923/real 1787696923] req@ffff9d85c0fb3800 x1874531134330368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696923.283750:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134330368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696923.283760:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696923.283762:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696923.283764:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134330368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696923.283777:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134330496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696923.283787:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696923.283792:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696923/real 1787696923] req@ffff9d85c0fb1c00 x1874531134330496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696923.283800:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134330496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696923.283808:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696923.283812:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696923.283815:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134330496:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696923.859037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696924.883123:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696925.907062:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696926.931015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696927.954994:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696928.402930:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696928.402939:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696928.402972:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696928.402981:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4830 00000100:00080000:3.0:1787696928.402990:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696928.402996:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696928.403030:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134330624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696928.403057:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696928.403059:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696928.403061:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696928.403064:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4830 00000100:00080000:3.0:1787696928.403066:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696928.403083:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696928.403092:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696928.403101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134330752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696928.403110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696928.403112:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696928.403115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696928.403118:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134330624:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696928.403119:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4830 00000100:00080000:3.0:1787696928.403124:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696928.403128:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696928.403132:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696928.403139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134330880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696928.403146:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696928.403149:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696928.403150:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696928.403152:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4830 00000100:00080000:3.0:1787696928.403155:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696928.403157:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696928.403160:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696928.403168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134331008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696928.403174:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696928.403177:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696928.403178:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696928.403180:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4830 00000100:00080000:3.0:1787696928.403183:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696928.403187:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696928.403191:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696928.403196:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696928.403204:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134331136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696928.403210:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696928.403212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696928.403213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696928.403217:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4830 00000100:00080000:3.0:1787696928.403221:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696928.403226:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403230:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696928/real 1787696928] req@ffff9d85c0fb2d80 x1874531134330624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696928.403233:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696928.403253:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134330624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696928.403255:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134331264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696928.403272:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696928.403277:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696928.403278:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696928.403285:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134330624:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696928.403287:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696928.403292:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696928.403297:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4830 00000100:00080000:3.0:1787696928.403301:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696928.403305:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403310:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134330752:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787696928.403314:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696928.403331:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134331392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696928.403332:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696928/real 1787696928] req@ffff9d85c0fb0380 x1874531134330752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696928.403356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134330752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696928.403361:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696928.403364:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696928.403368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134330752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696928.403373:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134330880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696928.403384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696928/real 1787696928] req@ffff9d85c0fb3b80 x1874531134330880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696928.403397:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134330880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696928.403402:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696928.403404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696928.403406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134330880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696928.403409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696928.403449:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696928/real 1787696928] req@ffff9d85c0fb1c00 x1874531134331008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696928.403476:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134331008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696928.403493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696928.403495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696928.403497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696928.403501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134331136:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696928.403523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696928/real 1787696928] req@ffff9d85c0fb3800 x1874531134331136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696928.403539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134331136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696928.403544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696928.403546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696928.403548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134331136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696928.403551:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134331264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696928.403561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696928/real 1787696928] req@ffff9d85c0fb0e00 x1874531134331264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696928.403574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134331264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696928.403579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696928.403581:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696928.403583:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134331264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696928.403595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134331392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696928.403605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696928.403610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696928/real 1787696928] req@ffff9d85c0fb0a80 x1874531134331392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696928.403617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134331392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696928.403622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696928.403624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696928.403626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134331392:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696928.979075:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696930.003012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696931.027029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696932.051040:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696933.075087:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696933.522919:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696933.522930:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696933.522935:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696933.522945:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4835 00000100:00080000:3.0:1787696933.522957:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696933.522965:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696933.523010:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134331520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696933.523036:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696933.523045:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696933.523047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696933.523052:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4835 00000100:00080000:3.0:1787696933.523056:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696933.523061:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696933.523073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696933.523084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134331520:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696933.523091:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134331648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696933.523106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696933.523110:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696933.523113:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696933.523118:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4835 00000100:00080000:3.0:1787696933.523123:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696933.523128:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696933.523134:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696933.523142:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696933.523148:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134331776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696933.523160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696933.523164:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696933.523167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696933.523171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4835 00000100:00080000:3.0:1787696933.523175:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696933.523176:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696933/real 1787696933] req@ffff9d85c0fb1500 x1874531134331520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696933.523183:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696933.523192:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696933.523206:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134331520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696933.523208:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134331904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696933.523239:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696933.523242:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696933.523243:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696933.523247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696933.523253:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696933.523255:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4835 00000100:00080000:3.0:1787696933.523261:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696933.523265:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134331520:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696933.523268:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696933.523277:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696933.523288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134332032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696933.523297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331648:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696933.523302:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696933.523308:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696933.523310:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696933.523316:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4835 00000100:00080000:3.0:1787696933.523321:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696933.523323:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696933.523328:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696933.523350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696933/real 1787696933] req@ffff9d85c0fb2680 x1874531134331648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696933.523357:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696933.523368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134331648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696933.523375:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134332160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696933.523390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696933.523395:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696933.523398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696933.523402:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696933.523406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331648:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696933.523407:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696933.523416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331776:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787696933.523417:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4835 00000100:00080000:3.0:1787696933.523425:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696933.523430:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696933.523437:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696933.523442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696933.523453:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134332288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696933.523456:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696933/real 1787696933] req@ffff9d85c0fb3100 x1874531134331776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696933.523484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134331776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696933.523491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696933.523493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696933.523496:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696933.523503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696933.523515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696933.523522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696933/real 1787696933] req@ffff9d85c0fb3480 x1874531134331904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696933.523529:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134331904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696933.523534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696933.523536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696933.523538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134331904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696933.523548:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332032:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696933.523559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696933.523565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696933/real 1787696933] req@ffff9d85c0fb2300 x1874531134332032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696933.523573:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134332032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696933.523578:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696933.523579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696933.523581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696933.523586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696933.523603:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696933.523614:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696933/real 1787696933] req@ffff9d85c0fb1880 x1874531134332160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696933.523626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134332160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696933.523635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696933.523652:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696933.523656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696933.523663:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696933.523684:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696933.523695:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696933/real 1787696933] req@ffff9d85c0fb2a00 x1874531134332288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696933.523703:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134332288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696933.523708:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696933.523710:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696933.523712:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332288:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696934.099088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696935.122952:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696936.146985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696937.170982:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696938.194981:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696938.642968:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696938.642976:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696938.642980:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696938.642988:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4840 00000100:00080000:3.0:1787696938.642998:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696938.643005:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696938.643037:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134332416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696938.643060:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696938.643063:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696938.643064:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696938.643067:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4840 00000100:00080000:3.0:1787696938.643069:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696938.643072:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696938.643080:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696938.643088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134332544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696938.643095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696938.643098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696938.643099:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696938.643102:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4840 00000100:00080000:3.0:1787696938.643104:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696938.643106:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696938.643109:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696938.643114:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134332416:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696938.643118:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134332672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696938.643129:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696938.643132:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696938.643133:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696938.643135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4840 00000100:00080000:3.0:1787696938.643137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696938.643140:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696938.643142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696938.643150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134332800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696938.643156:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696938.643158:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696938.643159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696938.643162:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4840 00000100:00080000:3.0:1787696938.643165:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696938.643167:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696938.643169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696938.643175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134332928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696938.643181:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696938.643184:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696938.643185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696938.643191:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696938.643192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4840 00000100:00080000:3.0:1787696938.643197:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696938.643200:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696938.643203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696938.643211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134333056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696938.643217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696938.643219:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696938.643220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696938.643223:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4840 00000100:00080000:3.0:1787696938.643225:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696938.643227:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696938.643231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696938.643236:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134333184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696938.643273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696938/real 1787696938] req@ffff9d85c0fb0700 x1874531134332416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696938.643297:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134332416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696938.643308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696938.643313:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696938.643318:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134332416:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696938.643327:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134332544:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696938.643353:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696938.643368:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696938/real 1787696938] req@ffff9d85c0fb0000 x1874531134332544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696938.643382:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134332544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696938.643392:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696938.643396:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696938.643400:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134332544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696938.643426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134332672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696938.643450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696938.643463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696938/real 1787696938] req@ffff9d85c0fb1f80 x1874531134332672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696938.643476:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134332672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696938.643485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696938.643489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696938.643493:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134332672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696938.643500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134332800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696938.643522:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696938.643536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696938/real 1787696938] req@ffff9d85c0fb2a00 x1874531134332800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696938.643549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134332800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696938.643558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696938.643561:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696938.643565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134332800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696938.643571:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332928:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696938.643611:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696938.643625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696938/real 1787696938] req@ffff9d85c0fb1880 x1874531134332928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696938.643646:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134332928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696938.643654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696938.643658:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696938.643662:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134332928:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696938.643668:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696938.643689:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696938.643701:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696938/real 1787696938] req@ffff9d85c0fb2300 x1874531134333056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696938.643714:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134333056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696938.643723:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696938.643727:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696938.643730:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696938.643737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696938.643757:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696938.643769:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696938/real 1787696938] req@ffff9d85c0fb3480 x1874531134333184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696938.643781:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134333184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696938.643790:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696938.643793:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696938.643797:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333184:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787696939.219060:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696940.243051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696941.267042:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696942.291043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696943.314994:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696943.762894:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696943.762902:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696943.762906:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696943.762914:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4845 00000100:00080000:3.0:1787696943.762923:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696943.762929:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696943.762960:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134333312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696943.762985:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696943.762988:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696943.762993:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696943.762997:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4845 00000100:00080000:3.0:1787696943.762999:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696943.763002:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696943.763010:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696943.763020:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134333440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696943.763027:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696943.763029:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696943.763031:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696943.763033:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4845 00000100:00080000:3.0:1787696943.763035:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696943.763037:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696943.763042:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696943.763048:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134333312:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696943.763050:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134333568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696943.763062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696943.763065:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696943.763066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696943.763068:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4845 00000100:00080000:3.0:1787696943.763070:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696943.763073:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696943.763076:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696943.763083:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134333696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696943.763090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696943.763093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696943.763094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696943.763096:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4845 00000100:00080000:3.0:1787696943.763098:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696943.763101:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696943.763104:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696943.763112:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134333824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696943.763117:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696943.763125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696943.763128:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696943.763130:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696943.763134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4845 00000100:00080000:3.0:1787696943.763137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696943.763140:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696943.763143:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696943.763150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134333952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696943.763168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696943.763171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696943.763172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696943.763176:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4845 00000100:00080000:3.0:1787696943.763179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696943.763182:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696943.763187:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696943/real 1787696943] req@ffff9d85c0fb3100 x1874531134333312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696943.763189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696943.763202:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134334080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696943.763216:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134333312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696943.763230:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696943.763234:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696943.763238:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134333312:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696943.763247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134333440:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696943.763274:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696943.763285:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696943/real 1787696943] req@ffff9d85c0fb2680 x1874531134333440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696943.763294:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134333440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696943.763308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696943.763311:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696943.763313:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134333440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696943.763332:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134333568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696943.763346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696943.763354:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696943/real 1787696943] req@ffff9d85c0fb1500 x1874531134333568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696943.763362:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134333568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696943.763367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696943.763369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696943.763372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134333568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696943.763375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134333696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696943.763385:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696943.763391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696943/real 1787696943] req@ffff9d85c0fb0a80 x1874531134333696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696943.763398:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134333696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696943.763404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696943.763405:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696943.763407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134333696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696943.763410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333824:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696943.763420:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696943.763425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696943/real 1787696943] req@ffff9d85c0fb0e00 x1874531134333824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696943.763432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134333824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696943.763437:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696943.763439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696943.763441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333824:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696943.763444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696943.763454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696943.763460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696943/real 1787696943] req@ffff9d85c0fb3800 x1874531134333952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696943.763466:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134333952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696943.763476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696943.763477:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696943.763479:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134333952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696943.763482:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696943.763491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696943.763496:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696943/real 1787696943] req@ffff9d85c0fb1c00 x1874531134334080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696943.763513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134334080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696943.763517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696943.763519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696943.763521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334080:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787696944.339035:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696945.362988:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696946.386981:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696947.411000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696948.435009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696948.882966:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696948.882976:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696948.882979:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696948.882988:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4850 00000100:00080000:3.0:1787696948.882997:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696948.883004:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696948.883037:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134334208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696948.883062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696948.883065:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696948.883078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696948.883083:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4850 00000100:00080000:3.0:1787696948.883086:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696948.883090:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696948.883098:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696948.883110:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134334336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696948.883118:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134334208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696948.883124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696948.883129:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696948.883130:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696948.883135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4850 00000100:00080000:3.0:1787696948.883138:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696948.883141:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696948.883145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696948.883156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134334464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696948.883165:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696948.883168:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696948.883170:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696948.883174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4850 00000100:00080000:3.0:1787696948.883177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696948.883180:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696948.883183:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696948.883189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696948.883200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134334592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696948.883209:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696948.883212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696948.883213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696948.883217:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4850 00000100:00080000:3.0:1787696948.883220:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696948.883221:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696948/real 1787696948] req@ffff9d85c0fb3b80 x1874531134334208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696948.883226:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696948.883236:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696948.883242:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134334208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696948.883248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134334720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696948.883252:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696948.883260:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696948.883264:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696948.883266:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134334208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696948.883271:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696948.883274:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696948.883278:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4850 00000100:00080000:3.0:1787696948.883281:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696948.883285:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696948.883288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134334336:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787696948.883291:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696948.883306:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134334848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696948.883310:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696948.883321:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696948/real 1787696948] req@ffff9d85c0fb0380 x1874531134334336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696948.883321:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696948.883330:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696948.883335:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696948.883338:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134334336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696948.883341:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4850 00000100:00080000:3.0:1787696948.883348:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696948.883350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696948.883354:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787696948.883355:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696948.883360:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134334336:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787696948.883363:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696948.883366:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134334464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696948.883379:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134334976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696948.883383:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696948.883392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696948/real 1787696948] req@ffff9d85c0fb2d80 x1874531134334464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696948.883401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134334464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696948.883405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696948.883407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696948.883410:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134334464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696948.883413:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134334592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696948.883424:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696948.883430:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696948/real 1787696948] req@ffff9d85c0fb1c00 x1874531134334592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696948.883436:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134334592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696948.883441:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696948.883442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696948.883444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134334592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696948.883447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334720:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696948.883455:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696948.883461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696948/real 1787696948] req@ffff9d85c0fb3800 x1874531134334720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696948.883467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134334720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696948.883471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696948.883473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696948.883474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696948.883483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696948.883492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696948.883500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696948/real 1787696948] req@ffff9d85c0fb0e00 x1874531134334848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696948.883510:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134334848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696948.883517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696948.883520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696948.883523:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696948.883529:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696948.883545:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696948.883551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696948/real 1787696948] req@ffff9d85c0fb0a80 x1874531134334976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696948.883558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134334976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696948.883563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696948.883564:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696948.883566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134334976:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696949.459019:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696950.482975:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696951.507045:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696952.530984:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696953.555013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787696954.002951:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696954.002959:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696954.002962:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696954.002969:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4855 00000100:00080000:0.0:1787696954.002994:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696954.003002:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787696954.003032:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134335104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696954.003056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696954.003058:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696954.003060:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696954.003062:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4855 00000100:00080000:0.0:1787696954.003065:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696954.003066:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696954.003073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696954.003081:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134335232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696954.003087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696954.003089:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696954.003090:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696954.003092:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4855 00000100:00080000:0.0:1787696954.003094:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696954.003097:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696954.003101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696954.003103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134335104:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787696954.003108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134335360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696954.003116:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696954.003118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696954.003119:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696954.003121:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4855 00000100:00080000:0.0:1787696954.003123:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696954.003124:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696954.003126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696954.003133:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134335488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696954.003138:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696954.003140:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696954.003141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696954.003143:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4855 00000100:00080000:0.0:1787696954.003161:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696954.003165:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696954.003169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696954.003170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787696954.003184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134335616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696954.003194:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696954.003198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696954.003200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696954.003204:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4855 00000100:00080000:0.0:1787696954.003208:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696954.003210:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696954/real 1787696954] req@ffff9d85b9d0b480 x1874531134335104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696954.003214:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696954.003221:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696954.003230:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134335104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696954.003236:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134335744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696954.003239:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.003246:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696954.003252:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134335104:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787696954.003253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696954.003258:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696954.003258:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134335232:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787696954.003263:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696954.003270:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4855 00000100:00080000:0.0:1787696954.003276:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696954.003279:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787696954.003283:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787696954.003288:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696954.003293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696954/real 1787696954] req@ffff9d85b9d08e00 x1874531134335232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696954.003299:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134335872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696954.003322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134335232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.003332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.003336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696954.003339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134335232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696954.003363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134335360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696954.003382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696954.003390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696954/real 1787696954] req@ffff9d85b9d08a80 x1874531134335360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.003398:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134335360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.003405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.003408:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696954.003411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134335360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696954.003416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134335488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696954.003434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696954.003445:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696954/real 1787696954] req@ffff9d85b9d08000 x1874531134335488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.003453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134335488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.003458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.003460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696954.003461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134335488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696954.003465:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134335616:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696954.003476:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696954.003482:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696954/real 1787696954] req@ffff9d85b9d0bb80 x1874531134335616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.003488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134335616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.003494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.003497:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696954.003500:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134335616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696954.003505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134335744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696954.003521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696954.003531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696954/real 1787696954] req@ffff9d85b9d0ad80 x1874531134335744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.003542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134335744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.003549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.003551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696954.003554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134335744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696954.003559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134335872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696954.003576:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696954.003586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696954/real 1787696954] req@ffff9d85b9d0a680 x1874531134335872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.003597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134335872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.003605:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.003608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696954.003611:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134335872:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696954.579009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696955.603036:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696956.626970:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696957.650992:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696958.675002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696959.122961:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696959.122968:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696959.122972:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696959.122980:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4860 00000100:00080000:3.0:1787696959.122990:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696959.122995:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696959.123031:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134336000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696959.123055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696959.123057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696959.123058:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696959.123061:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4860 00000100:00080000:3.0:1787696959.123063:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696959.123065:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696959.123073:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696959.123081:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134336128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696959.123087:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696959.123089:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696959.123090:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696959.123092:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4860 00000100:00080000:3.0:1787696959.123094:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696959.123096:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696959.123099:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696959.123101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134336000:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696959.123106:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134336256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696959.123115:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696959.123117:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696959.123118:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696959.123120:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4860 00000100:00080000:3.0:1787696959.123122:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696959.123123:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696959.123126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696959.123132:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134336384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696959.123138:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696959.123140:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696959.123141:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696959.123143:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4860 00000100:00080000:3.0:1787696959.123145:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696959.123148:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696959.123150:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696959.123156:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134336512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696959.123161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696959.123163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696959.123166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696959.123167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696959.123170:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4860 00000100:00080000:3.0:1787696959.123175:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696959.123178:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696959.123181:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696959.123187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134336640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696959.123192:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696959.123195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696959.123196:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696959.123198:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4860 00000100:00080000:3.0:1787696959.123201:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696959.123204:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696959/real 1787696959] req@ffff9d85c0fb1500 x1874531134336000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696959.123205:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696959.123211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696959.123223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134336768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696959.123226:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134336000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696959.123236:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696959.123240:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696959.123244:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134336000:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696959.123250:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134336128:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696959.123266:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696959.123273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696959/real 1787696959] req@ffff9d85c0fb2680 x1874531134336128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696959.123281:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134336128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696959.123285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696959.123287:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696959.123289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134336128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696959.123306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134336256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696959.123316:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696959.123321:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696959/real 1787696959] req@ffff9d85c0fb3100 x1874531134336256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696959.123327:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134336256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696959.123332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696959.123334:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696959.123335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134336256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696959.123338:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134336384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696959.123346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696959.123351:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696959/real 1787696959] req@ffff9d85c0fb3480 x1874531134336384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696959.123369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134336384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696959.123377:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696959.123380:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696959.123382:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134336384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696959.123388:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134336512:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696959.123406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696959.123417:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696959/real 1787696959] req@ffff9d85c0fb2300 x1874531134336512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696959.123426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134336512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696959.123433:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696959.123436:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696959.123438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134336512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696959.123443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134336640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696959.123458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696959.123468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696959/real 1787696959] req@ffff9d85c0fb1880 x1874531134336640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696959.123477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134336640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696959.123520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696959.123523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696959.123525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134336640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696959.123530:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134336768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696959.123547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696959.123557:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696959/real 1787696959] req@ffff9d85c0fb2a00 x1874531134336768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696959.123567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134336768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696959.123574:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696959.123576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696959.123578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134336768:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696959.699002:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696960.723050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696961.747058:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696962.770977:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696963.795027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696964.242974:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696964.242984:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696964.242988:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696964.242999:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4866 00000100:00080000:3.0:1787696964.243011:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696964.243018:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696964.243061:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134336896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696964.243094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696964.243100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696964.243102:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696964.243107:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4866 00000100:00080000:3.0:1787696964.243111:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696964.243116:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696964.243126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696964.243137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134337024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696964.243143:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134336896:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696964.243148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696964.243151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696964.243153:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696964.243157:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4866 00000100:00080000:3.0:1787696964.243160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696964.243164:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696964.243170:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696964.243182:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134337152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696964.243192:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696964.243195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696964.243197:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696964.243200:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4866 00000100:00080000:3.0:1787696964.243204:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696964.243207:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696964.243212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696964.243219:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696964.243224:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134337280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696964.243231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696964.243234:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696964.243235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696964.243238:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4866 00000100:00080000:3.0:1787696964.243241:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696964.243243:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696964.243246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696964.243252:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134337408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696964.243257:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696964.243260:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696964.243261:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696964.243263:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4866 00000100:00080000:3.0:1787696964.243265:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696964.243267:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696964.243269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696964.243276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134337536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696964.243281:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696964.243282:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696964.243287:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696964.243288:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696964/real 1787696964] req@ffff9d85c0fb1f80 x1874531134336896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696964.243299:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4866 00000100:00080000:3.0:1787696964.243310:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696964.243316:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696964.243320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134336896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696964.243323:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696964.243340:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134337664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696964.243344:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696964.243352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696964.243356:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134336896:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696964.243379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337024:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696964.243406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696964.243419:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696964/real 1787696964] req@ffff9d85c0fb0000 x1874531134337024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696964.243432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134337024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696964.243441:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696964.243444:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696964.243448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696964.243452:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696964.243467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696964.243475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696964/real 1787696964] req@ffff9d85c0fb0700 x1874531134337152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696964.243482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134337152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696964.243487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696964.243489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696964.243491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696964.243494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696964.243509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696964.243519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696964/real 1787696964] req@ffff9d85c0fb2a00 x1874531134337280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696964.243531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134337280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696964.243539:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696964.243542:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696964.243545:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696964.243550:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134337408:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696964.243564:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696964.243578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696964/real 1787696964] req@ffff9d85c0fb1880 x1874531134337408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696964.243585:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134337408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696964.243590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696964.243592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696964.243594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134337408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696964.243597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134337536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696964.243607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696964.243612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696964/real 1787696964] req@ffff9d85c0fb2300 x1874531134337536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696964.243618:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134337536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696964.243624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696964.243626:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696964.243628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134337536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696964.243630:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134337664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696964.243639:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696964.243643:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696964/real 1787696964] req@ffff9d85c0fb3480 x1874531134337664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696964.243652:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134337664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696964.243660:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696964.243663:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696964.243666:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134337664:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696964.819010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696965.843030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696966.866990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696967.891022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696968.915007:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696969.362937:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696969.362944:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696969.362947:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696969.362954:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4871 00000100:00080000:3.0:1787696969.362962:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696969.362969:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696969.362998:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134337792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696969.363019:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696969.363021:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696969.363023:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696969.363025:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4871 00000100:00080000:3.0:1787696969.363027:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696969.363029:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696969.363036:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696969.363044:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134337920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696969.363058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696969.363061:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696969.363063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696969.363068:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134337792:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696969.363070:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4871 00000100:00080000:3.0:1787696969.363076:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696969.363082:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696969.363087:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696969.363099:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134338048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696969.363108:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696969.363112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696969.363114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696969.363152:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696969.363153:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4871 00000100:00080000:3.0:1787696969.363160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696969.363165:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696969.363171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696969.363194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134338176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696969.363198:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696969/real 1787696969] req@ffff9d85c0fb3100 x1874531134337792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696969.363217:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696969.363220:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696969.363222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696969.363227:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4871 00000100:00100000:1.0:1787696969.363231:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134337792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696969.363233:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696969.363241:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696969.363245:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787696969.363245:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696969.363252:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134337792:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787696969.363252:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696969.363266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134338304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696969.363277:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337920:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787696969.363279:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696969.363284:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696969.363286:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696969.363291:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4871 00000100:00080000:3.0:1787696969.363296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696969.363297:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696969.363304:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696969.363309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696969/real 1787696969] req@ffff9d85c0fb2680 x1874531134337920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696969.363310:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696969.363323:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134337920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696969.363330:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696969.363330:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134338432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696969.363336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696969.363339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134337920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696969.363343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338048:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787696969.363354:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696969.363357:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696969.363359:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696969.363362:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4871 00000100:00080000:3.0:1787696969.363365:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696969.363370:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787696969.363372:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787696969.363374:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696969.363385:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134338560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696969.363389:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696969/real 1787696969] req@ffff9d85c0fb1500 x1874531134338048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696969.363418:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134338048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696969.363428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696969.363431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696969.363435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696969.363442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696969.363461:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696969.363473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696969/real 1787696969] req@ffff9d85c0fb0a80 x1874531134338176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696969.363485:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134338176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696969.363493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696969.363496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696969.363499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696969.363514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134338304:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696969.363533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696969.363543:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696969/real 1787696969] req@ffff9d85c0fb0e00 x1874531134338304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696969.363555:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134338304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696969.363560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696969.363563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696969.363565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134338304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696969.363568:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134338432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696969.363581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696969.363587:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696969/real 1787696969] req@ffff9d85c0fb3800 x1874531134338432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696969.363593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134338432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696969.363598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696969.363599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696969.363601:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134338432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696969.363604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134338560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696969.363611:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696969.363616:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696969/real 1787696969] req@ffff9d85c0fb1c00 x1874531134338560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696969.363622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134338560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696969.363626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696969.363628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696969.363629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134338560:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696969.939055:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696970.963106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696971.987013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696973.011006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696974.034989:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696974.482941:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696974.482948:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696974.482951:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696974.482958:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4876 00000100:00080000:3.0:1787696974.482967:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696974.482973:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787696974.483002:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134338688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696974.483021:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696974.483023:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696974.483025:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696974.483027:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4876 00000100:00080000:3.0:1787696974.483029:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696974.483031:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696974.483038:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696974.483044:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134338816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696974.483051:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696974.483053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696974.483054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696974.483056:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4876 00000100:00080000:3.0:1787696974.483058:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696974.483060:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696974.483064:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696974.483069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134338944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696974.483077:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696974.483079:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696974.483080:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696974.483083:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4876 00000100:00100000:1.0:1787696974.483084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134338688:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787696974.483084:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696974.483088:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696974.483092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696974.483100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134339072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696974.483105:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696974.483107:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696974.483108:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696974.483110:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4876 00000100:00080000:3.0:1787696974.483111:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696974.483113:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696974.483116:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696974.483123:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134339200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696974.483128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696974.483130:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696974.483131:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696974.483133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4876 00000100:00080000:3.0:1787696974.483135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696974.483138:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696974.483141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696974.483146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134339328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696974.483150:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787696974.483156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696974.483159:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696974.483160:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696974.483163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4876 00000100:00080000:3.0:1787696974.483166:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696974.483168:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787696974.483171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696974.483177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134339456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696974.483196:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696974/real 1787696974] req@ffff9d85c0fb2d80 x1874531134338688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696974.483210:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134338688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696974.483217:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696974.483221:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696974.483224:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134338688:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787696974.483229:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338816:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696974.483243:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696974.483250:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696974/real 1787696974] req@ffff9d85c0fb0380 x1874531134338816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696974.483257:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134338816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696974.483261:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696974.483263:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696974.483264:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696974.483281:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696974.483290:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696974.483296:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696974/real 1787696974] req@ffff9d85c0fb3b80 x1874531134338944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696974.483302:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134338944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696974.483306:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696974.483308:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696974.483309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134338944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696974.483313:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696974.483321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696974.483325:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696974/real 1787696974] req@ffff9d85c0fb1c00 x1874531134339072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696974.483332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134339072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696974.483336:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696974.483338:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696974.483339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696974.483342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134339200:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787696974.483350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696974.483355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696974/real 1787696974] req@ffff9d85c0fb3800 x1874531134339200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696974.483360:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134339200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696974.483364:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696974.483366:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696974.483368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134339200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696974.483371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134339328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696974.483378:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696974.483383:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696974/real 1787696974] req@ffff9d85c0fb0e00 x1874531134339328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696974.483388:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134339328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696974.483396:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696974.483398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696974.483400:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134339328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787696974.483402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134339456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787696974.483410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787696974.483414:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696974/real 1787696974] req@ffff9d85c0fb0a80 x1874531134339456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696974.483419:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134339456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696974.483423:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696974.483425:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787696974.483427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134339456:192.168.201.137@tcp:8:kworker.0 00010000:00010000:0.0:1787696975.058979:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696976.083030:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696977.106998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696978.131066:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696979.155067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696979.603018:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696979.603031:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696979.603036:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696979.603048:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4881 00000100:00080000:1.0:1787696979.603062:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696979.603071:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696979.603111:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134339584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696979.603137:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696979.603140:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696979.603141:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696979.603144:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4881 00000100:00080000:1.0:1787696979.603147:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696979.603150:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696979.603159:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696979.603168:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134339712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696979.603172:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134339584:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787696979.603179:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696979.603182:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696979.603183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696979.603186:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4881 00000100:00080000:1.0:1787696979.603188:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696979.603190:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696979.603194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696979.603201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134339840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696979.603207:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696979.603209:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696979.603210:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696979.603212:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4881 00000100:00080000:1.0:1787696979.603214:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696979.603216:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696979.603220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696979.603227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134339968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696979.603235:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696979.603239:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696979.603240:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696979.603244:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4881 00000100:00080000:1.0:1787696979.603248:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696979.603253:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696979.603258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696979.603271:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134340096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696979.603281:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696979.603284:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696979.603285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696979.603287:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339712:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787696979.603291:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4881 00000100:00080000:1.0:1787696979.603296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696979.603299:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696979.603303:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696979.603314:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134340224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696979.603320:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339840:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787696979.603337:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787696979.603343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339968:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787696979.603348:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696979.603352:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696979.603357:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4881 00000100:00080000:1.0:1787696979.603362:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696979.603366:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696979.603372:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696979.603377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787696979.603387:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134340352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696979.603416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696979.603443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340352:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696980.179037:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696981.203052:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696982.227026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787696982.675174:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787696982.675233:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787696982.675255:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696982.675314:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696982.675326:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696982.675334:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696982.675343:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696982.675351:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787696982.675370:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787696982.675451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696979/real 1787696982] req@ffff9d85b9d2ca80 x1874531134340224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696982.675480:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134340224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696982.675518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696982.675524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696982.675529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696982.675537:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696979/real 1787696982] req@ffff9d85b9d2f480 x1874531134340352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696982.675549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134340352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696982.675557:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696982.675571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696982.675574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696982.675580:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696979/real 1787696982] req@ffff9d85b9d2c000 x1874531134339712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696982.675591:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134339712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696982.675603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696982.675606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696982.675609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696982.675616:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696979/real 1787696982] req@ffff9d85b9d2ce00 x1874531134339840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696982.675626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134339840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696982.675633:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696982.675636:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696982.675639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696982.675646:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696979/real 1787696982] req@ffff9d85b9d2e300 x1874531134339968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696982.675657:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134339968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696982.675665:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696982.675668:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696982.675672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134339968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696982.675679:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696979/real 1787696982] req@ffff9d85b9d2fb80 x1874531134340096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696982.675691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134340096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696982.675698:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696982.675702:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696982.675705:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696982.675710:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696979/real 1787696982] req@ffff9d85b9d2c380 x1874531134339584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696982.675721:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134339584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696982.675727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696982.675730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696982.675733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134339584:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787696983.251043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696984.274998:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696984.722975:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696984.722985:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696984.722989:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696984.722999:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4886 00000100:00080000:1.0:1787696984.723010:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696984.723016:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696984.723053:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134340480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696984.723092:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696984.723096:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696984.723098:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696984.723102:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4886 00000100:00080000:1.0:1787696984.723106:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696984.723109:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696984.723118:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696984.723131:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134340608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696984.723146:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696984.723149:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696984.723150:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696984.723154:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4886 00000100:00080000:1.0:1787696984.723157:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696984.723161:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696984.723165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696984.723173:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134340736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696984.723189:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696984.723192:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696984.723193:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696984.723196:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4886 00000100:00080000:1.0:1787696984.723200:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696984.723204:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696984.723208:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696984.723219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134340864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696984.723227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696984.723229:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696984.723230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696984.723233:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4886 00000100:00080000:1.0:1787696984.723235:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696984.723238:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696984.723241:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696984.723247:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134340992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696984.723252:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696984.723254:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696984.723255:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696984.723257:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4886 00000100:00080000:1.0:1787696984.723258:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696984.723260:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696984.723262:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696984.723266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134341120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696984.723271:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696984.723272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696984.723273:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696984.723275:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4886 00000100:00080000:1.0:1787696984.723277:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696984.723279:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696984.723280:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696984.723286:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134341248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696984.723541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134340480:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696984.723598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696984.723653:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696984/real 1787696984] req@ffff9d85b9d2e680 x1874531134340480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696984.723669:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134340480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696984.723678:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696984.723695:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696984.723698:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134340480:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696984.723703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134340608:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696984.723726:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696984.723739:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696984/real 1787696984] req@ffff9d85b9d2c700 x1874531134340608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696984.723751:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134340608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696984.723759:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696984.723762:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696984.723764:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134340608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696984.723769:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134340736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696984.723789:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696984.723814:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696984/real 1787696984] req@ffff9d85b9d2ed80 x1874531134340736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696984.723824:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134340736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696984.723832:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696984.723835:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696984.723842:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134340736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696984.723848:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134340864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696984.723889:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696984.723913:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696984/real 1787696984] req@ffff9d85b9d2d500 x1874531134340864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696984.723931:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134340864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696984.723938:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696984.723941:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696984.723945:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134340864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696984.723962:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340992:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696984.723983:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696984.724000:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696984/real 1787696984] req@ffff9d85b9d2f800 x1874531134340992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696984.724012:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134340992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696984.724019:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696984.724022:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696984.724025:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134340992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696984.724030:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134341120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696984.724050:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696984.724060:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696984/real 1787696984] req@ffff9d85b9d2dc00 x1874531134341120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696984.724078:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134341120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696984.724090:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696984.724094:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696984.724096:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134341120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696984.724101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134341248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696984.724120:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696984.724129:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696984/real 1787696984] req@ffff9d85b9d2c380 x1874531134341248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696984.724139:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134341248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696984.724146:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696984.724148:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696984.724151:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134341248:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696985.299032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696986.323060:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696987.347030:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696988.371080:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696989.394986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696989.842949:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696989.842956:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696989.842959:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696989.842967:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4891 00000100:00080000:1.0:1787696989.842977:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696989.842982:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787696989.843013:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134341376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696989.843046:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696989.843050:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696989.843052:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696989.843055:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4891 00000100:00080000:1.0:1787696989.843058:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696989.843062:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696989.843071:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696989.843084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134341504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696989.843093:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696989.843096:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696989.843098:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696989.843102:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4891 00000100:00080000:1.0:1787696989.843105:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696989.843108:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696989.843111:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696989.843120:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134341632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696989.843126:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696989.843128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696989.843129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696989.843131:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4891 00000100:00080000:1.0:1787696989.843132:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696989.843134:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696989.843136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696989.843144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134341760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696989.843152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696989.843155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696989.843156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696989.843160:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4891 00000100:00080000:1.0:1787696989.843163:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696989.843166:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696989.843171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696989.843180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134341888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696989.843187:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696989.843189:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696989.843190:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696989.843193:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4891 00000100:00080000:1.0:1787696989.843195:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696989.843197:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696989.843200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696989.843208:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134342016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696989.843215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696989.843218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696989.843220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696989.843223:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4891 00000100:00080000:1.0:1787696989.843226:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696989.843229:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787696989.843233:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696989.843244:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134342144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696989.843325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134341376:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787696989.843430:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696989.843513:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696989/real 1787696989] req@ffff9d85b9d2fb80 x1874531134341376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696989.843525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134341376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696989.843532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696989.843535:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696989.843537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134341376:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787696989.843542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134341504:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696989.843558:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696989.843566:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696989/real 1787696989] req@ffff9d85b9d2e300 x1874531134341504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696989.843573:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134341504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696989.843577:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696989.843579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696989.843581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134341504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696989.843584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134341632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696989.843593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696989.843599:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696989/real 1787696989] req@ffff9d85b9d2ce00 x1874531134341632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696989.843605:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134341632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696989.843609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696989.843611:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696989.843612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134341632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696989.843615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134341760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696989.843623:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696989.843628:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696989/real 1787696989] req@ffff9d85b9d2c000 x1874531134341760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696989.843635:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134341760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696989.843639:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696989.843640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696989.843642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134341760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696989.843645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134341888:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696989.843653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696989.843659:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696989/real 1787696989] req@ffff9d85b9d2f480 x1874531134341888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696989.843665:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134341888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696989.843669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696989.843670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696989.843672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134341888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696989.843674:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696989.843683:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696989.843687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696989/real 1787696989] req@ffff9d85b9d2ca80 x1874531134342016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696989.843692:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134342016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696989.843701:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696989.843702:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696989.843704:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696989.843707:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696989.843714:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696989.843719:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696989/real 1787696989] req@ffff9d85b9d2c380 x1874531134342144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696989.843724:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134342144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696989.843728:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696989.843730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696989.843731:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342144:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696990.419072:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696991.443119:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696992.467073:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696993.491046:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696994.515544:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696994.963026:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696994.963040:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696994.963045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696994.963057:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4896 00000100:00080000:2.0:1787696994.963070:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696994.963078:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787696994.963124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134342272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696994.963158:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696994.963161:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696994.963163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696994.963168:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4896 00000100:00080000:2.0:1787696994.963171:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696994.963175:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696994.963185:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696994.963205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134342400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787696994.963214:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134342272:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787696994.963219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696994.963224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696994.963226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696994.963231:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4896 00000100:00080000:2.0:1787696994.963234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696994.963238:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696994.963245:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696994.963256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134342528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696994.963263:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696994.963265:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696994.963267:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696994.963269:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4896 00000100:00080000:2.0:1787696994.963305:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696994.963308:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696994.963312:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696994.963322:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134342656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787696994.963330:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787696994.963336:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696994.963340:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696994.963342:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696994.963347:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4896 00000100:00080000:2.0:1787696994.963352:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696994.963357:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696994.963363:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696994.963375:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134342784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696994.963384:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696994.963387:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696994.963390:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696994.963394:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4896 00000100:00080000:2.0:1787696994.963398:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787696994.963403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696994/real 1787696994] req@ffff9d85babd4000 x1874531134342272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696994.963406:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696994.963415:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696994.963429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134342272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696994.963431:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134342912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696994.963454:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696994.963455:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696994.963462:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696994.963463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787696994.963465:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787696994.963471:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134342272:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787696994.963473:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4896 00000100:00080000:2.0:1787696994.963480:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696994.963486:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787696994.963491:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696994.963503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134342400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787696994.963504:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134343040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787696994.963533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696994.963548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696994/real 1787696994] req@ffff9d85babd6680 x1874531134342400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696994.963561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134342400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696994.963570:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696994.963573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696994.963576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134342400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696994.963582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134342528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696994.963598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696994.963607:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696994/real 1787696994] req@ffff9d85babd6a00 x1874531134342528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696994.963618:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134342528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696994.963627:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696994.963630:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696994.963633:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134342528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696994.963639:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134342656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696994.963653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696994.963663:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696994/real 1787696994] req@ffff9d85babd5180 x1874531134342656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696994.963674:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134342656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696994.963682:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696994.963685:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696994.963688:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134342656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696994.963706:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342784:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787696994.963724:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696994.963734:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696994/real 1787696994] req@ffff9d85babd4a80 x1874531134342784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696994.963746:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134342784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696994.963926:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696994.963930:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696994.963934:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342784:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787696994.963940:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696994.963962:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696994.963979:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696994/real 1787696994] req@ffff9d85babd7b80 x1874531134342912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696994.963991:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134342912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696994.963999:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696994.964003:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696994.964006:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134342912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787696994.964024:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787696994.964042:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787696994.964054:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696994/real 1787696994] req@ffff9d85babd7100 x1874531134343040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696994.964066:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134343040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696994.964075:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696994.964078:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787696994.964081:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343040:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787696995.539040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696996.564390:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696997.587054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787696998.611047:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787696999.634991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697000.083055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697000.083064:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697000.083068:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697000.083076:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4901 00000100:00080000:1.0:1787697000.083086:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697000.083092:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697000.083125:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134343168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697000.083161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697000.083164:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697000.083166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697000.083170:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4901 00000100:00080000:1.0:1787697000.083174:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697000.083179:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697000.083206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697000.083219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134343296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697000.083224:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134343168:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697000.083246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697000.083250:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697000.083252:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697000.083257:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4901 00000100:00080000:1.0:1787697000.083260:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697000.083266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697000.083271:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697000.083283:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134343424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697000.083296:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697000.083299:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697000.083300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697000.083302:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697000.083306:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4901 00000100:00080000:1.0:1787697000.083311:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697000.083315:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697000.083320:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697000.083330:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134343552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697000.083339:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697000.083346:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697000/real 1787697000] req@ffff9d85b9d2dc00 x1874531134343168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697000.083354:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697000.083359:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697000.083367:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4901 00000100:00080000:1.0:1787697000.083370:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697000.083374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134343168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697000.083378:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697000.083386:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697000.083391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697000.083399:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697000.083400:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134343680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697000.083405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134343168:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697000.083418:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697000.083422:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697000.083423:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697000.083427:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4901 00000100:00080000:1.0:1787697000.083430:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697000.083435:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697000.083437:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134343296:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787697000.083441:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697000.083455:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134343808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697000.083466:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697000.083477:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697000.083482:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787697000.083482:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697000/real 1787697000] req@ffff9d85b9d2f800 x1874531134343296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697000.083486:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697000.083493:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4901 00000100:00080000:1.0:1787697000.083498:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697000.083502:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697000.083506:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697000.083517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134343296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697000.083522:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134343936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697000.083533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697000.083540:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697000.083545:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134343296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697000.083552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134343424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697000.083572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697000.083582:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697000/real 1787697000] req@ffff9d85b9d2d500 x1874531134343424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697000.083594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134343424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697000.083601:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697000.083604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697000.083607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134343424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697000.083613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134343552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697000.083627:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697000.083643:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697000/real 1787697000] req@ffff9d85b9d2ed80 x1874531134343552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697000.083655:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134343552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697000.083662:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697000.083666:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697000.083669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134343552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697000.083673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343680:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697000.083689:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697000.083698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697000/real 1787697000] req@ffff9d85b9d2c700 x1874531134343680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697000.083721:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134343680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697000.083728:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697000.083731:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697000.083735:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697000.083749:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697000.083766:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697000.083786:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697000/real 1787697000] req@ffff9d85b9d2e680 x1874531134343808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697000.083797:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134343808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697000.083810:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697000.083813:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697000.083816:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697000.083821:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697000.083836:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697000.083870:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697000/real 1787697000] req@ffff9d85b9d2dc00 x1874531134343936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697000.083882:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134343936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697000.083890:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697000.083893:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697000.083896:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134343936:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697000.659029:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697001.682953:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697002.707007:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697003.731058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697004.755002:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697005.202963:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697005.202972:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697005.202975:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697005.202983:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4906 00000100:00080000:1.0:1787697005.202992:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697005.202998:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697005.203027:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134344064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697005.203049:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697005.203052:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697005.203053:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697005.203056:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4906 00000100:00080000:1.0:1787697005.203058:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697005.203060:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697005.203067:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697005.203075:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134344192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697005.203081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697005.203083:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697005.203085:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697005.203087:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4906 00000100:00080000:1.0:1787697005.203088:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697005.203091:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697005.203112:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697005.203124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134344320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697005.203131:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134344064:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697005.203139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697005.203143:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697005.203145:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697005.203149:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4906 00000100:00080000:1.0:1787697005.203152:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697005.203155:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697005.203161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697005.203174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134344448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697005.203183:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697005.203187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697005.203189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697005.203192:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4906 00000100:00080000:1.0:1787697005.203196:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697005.203199:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697005.203202:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697005.203207:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697005.203215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134344576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697005.203222:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697005.203224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697005.203226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697005.203228:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4906 00000100:00080000:1.0:1787697005.203231:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697005.203233:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697005.203235:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697005.203242:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134344704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697005.203249:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697005.203250:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697005/real 1787697005] req@ffff9d85b9d2c380 x1874531134344064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697005.203251:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697005.203254:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697005.203259:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4906 00000100:00080000:1.0:1787697005.203262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697005.203267:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697005.203271:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697005.203278:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134344064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697005.203279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134344832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697005.203308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697005.203312:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697005.203316:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134344064:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697005.203321:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134344192:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697005.203340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697005.203349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697005/real 1787697005] req@ffff9d85b9d2ca80 x1874531134344192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697005.203356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134344192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697005.203361:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697005.203363:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697005.203365:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134344192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697005.203382:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134344320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697005.203392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697005.203398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697005/real 1787697005] req@ffff9d85b9d2f480 x1874531134344320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697005.203405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134344320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697005.203409:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697005.203411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697005.203413:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134344320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697005.203416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134344448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697005.203424:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697005.203430:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697005/real 1787697005] req@ffff9d85b9d2c000 x1874531134344448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697005.203436:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134344448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697005.203441:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697005.203442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697005.203444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134344448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697005.203447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134344576:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697005.203456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697005.203461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697005/real 1787697005] req@ffff9d85b9d2ce00 x1874531134344576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697005.203468:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134344576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697005.203472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697005.203473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697005.203475:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134344576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697005.203478:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134344704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697005.203486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697005.203490:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697005/real 1787697005] req@ffff9d85b9d2e300 x1874531134344704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697005.203496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134344704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697005.203501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697005.203503:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697005.203504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134344704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697005.203507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134344832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697005.203515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697005.203519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697005/real 1787697005] req@ffff9d85b9d2fb80 x1874531134344832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697005.203525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134344832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697005.203529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697005.203530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697005.203532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134344832:192.168.201.137@tcp:8:kworker.0 00010000:00010000:0.0:1787697005.778955:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697006.803009:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697007.826999:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697008.851006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697009.875007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697010.322926:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697010.322933:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697010.322936:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697010.322944:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4912 00000100:00080000:2.0:1787697010.322952:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697010.322958:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697010.322988:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134344960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697010.323024:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697010.323027:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697010.323029:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697010.323032:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4912 00000100:00080000:2.0:1787697010.323035:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697010.323038:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697010.323046:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697010.323058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134345088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697010.323059:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134344960:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697010.323077:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697010.323080:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697010.323081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697010.323086:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4912 00000100:00080000:2.0:1787697010.323089:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697010.323092:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697010.323096:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697010.323103:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134345216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697010.323110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697010.323112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697010.323113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697010.323114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4912 00000100:00080000:2.0:1787697010.323116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697010.323121:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:3.0:1787697010.323121:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787697010.323126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697010.323134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134345344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697010.323141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697010.323143:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697010.323143:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697010.323145:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4912 00000100:00080000:2.0:1787697010.323147:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697010.323149:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697010.323152:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697010.323157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134345472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697010.323163:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697010.323167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787697010.323168:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697010/real 1787697010] req@ffff9d85babd4700 x1874531134344960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697010.323169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697010.323175:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4912 00000100:00080000:2.0:1787697010.323178:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697010.323184:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697010.323190:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697010.323196:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134345600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697010.323201:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134344960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697010.323204:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697010.323210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697010.323213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697010.323219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4912 00000100:00080000:2.0:1787697010.323223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697010.323223:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697010.323228:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697010.323231:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697010.323233:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697010.323237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134344960:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697010.323243:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134345728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697010.323257:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697010.323275:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697010.323283:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697010/real 1787697010] req@ffff9d85babd5f80 x1874531134345088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697010.323291:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134345088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697010.323295:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697010.323297:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697010.323299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697010.323302:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697010.323311:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697010.323317:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697010/real 1787697010] req@ffff9d85babd4e00 x1874531134345216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697010.323323:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134345216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697010.323327:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697010.323329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697010.323330:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697010.323333:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697010.323341:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697010.323347:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697010/real 1787697010] req@ffff9d85babd7100 x1874531134345344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697010.323353:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134345344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697010.323357:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697010.323358:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697010.323360:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697010.323363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134345472:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697010.323371:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697010.323375:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697010/real 1787697010] req@ffff9d85babd7b80 x1874531134345472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697010.323381:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134345472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697010.323385:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697010.323386:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697010.323388:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134345472:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697010.323391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134345600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697010.323398:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697010.323403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697010/real 1787697010] req@ffff9d85babd4a80 x1874531134345600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697010.323409:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134345600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697010.323414:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697010.323415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697010.323417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134345600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697010.323419:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134345728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697010.323427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697010.323432:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697010/real 1787697010] req@ffff9d85babd5180 x1874531134345728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697010.323437:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134345728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697010.323441:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697010.323443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697010.323444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134345728:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697010.899024:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697011.922989:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697012.947110:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697013.971002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697014.995119:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697015.443035:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697015.443050:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697015.443055:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697015.443069:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4917 00000100:00080000:1.0:1787697015.443083:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697015.443092:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697015.443144:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134345856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697015.443190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697015.443199:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697015.443202:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697015.443208:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4917 00000100:00080000:1.0:1787697015.443213:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697015.443218:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697015.443231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697015.443243:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134345856:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697015.443250:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134345984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697015.443265:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697015.443269:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697015.443272:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697015.443276:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4917 00000100:00080000:1.0:1787697015.443281:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697015.443287:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697015.443294:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697015.443308:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134346112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697015.443318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697015.443326:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697015.443331:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697015.443333:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697015.443339:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4917 00000100:00080000:1.0:1787697015.443344:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697015.443349:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697015.443355:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697015.443366:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697015/real 1787697015] req@ffff9d85b9d2fb80 x1874531134345856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697015.443371:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134346240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697015.443385:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697015.443388:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697015.443391:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697015.443392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134345856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697015.443408:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4917 00000100:00080000:3.0:1787697015.443409:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697015.443416:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697015.443418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697015.443426:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697015.443428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134345856:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787697015.443437:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697015.443453:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134346368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697015.443461:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345984:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697015.443463:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697015.443470:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697015.443472:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697015.443476:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4917 00000100:00080000:1.0:1787697015.443479:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697015.443483:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697015.443488:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697015.443493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697015.443498:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134346496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697015.443507:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697015.443511:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787697015.443512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697015/real 1787697015] req@ffff9d85b9d2e300 x1874531134345984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697015.443513:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697015.443520:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4917 00000100:00080000:1.0:1787697015.443526:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697015.443531:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697015.443534:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697015.443537:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134345984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697015.443545:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134346624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697015.443546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697015.443555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697015.443563:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134345984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697015.443571:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134346112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697015.443589:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697015.443596:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697015/real 1787697015] req@ffff9d85b9d2ce00 x1874531134346112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697015.443604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134346112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697015.443609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697015.443612:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697015.443614:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134346112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697015.443618:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134346240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697015.443628:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697015.443635:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697015/real 1787697015] req@ffff9d85b9d2c000 x1874531134346240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697015.443642:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134346240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697015.443656:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697015.443661:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697015.443663:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134346240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697015.443677:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134346368:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697015.443690:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697015.443697:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697015/real 1787697015] req@ffff9d85b9d2f480 x1874531134346368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697015.443705:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134346368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697015.443710:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697015.443712:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697015.443714:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134346368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697015.443724:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134346496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697015.443734:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697015.443740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697015/real 1787697015] req@ffff9d85b9d2ca80 x1874531134346496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697015.443748:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134346496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697015.443754:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697015.443757:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697015.443759:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134346496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697015.443762:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134346624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697015.443771:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697015.443777:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697015/real 1787697015] req@ffff9d85b9d2c380 x1874531134346624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697015.443784:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134346624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697015.443790:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697015.443791:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697015.443793:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134346624:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697016.019112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697017.043069:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697018.067069:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697019.090992:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697020.114976:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697020.563004:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697020.563020:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697020.563026:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697020.563040:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4922 00000100:00080000:1.0:1787697020.563053:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697020.563061:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697020.563115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134346752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697020.563150:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697020.563153:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697020.563154:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697020.563158:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4922 00000100:00080000:1.0:1787697020.563162:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697020.563165:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697020.563175:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697020.563186:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134346880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697020.563194:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697020.563197:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697020.563199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697020.563201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4922 00000100:00080000:1.0:1787697020.563203:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697020.563206:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697020.563210:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697020.563219:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134346752:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697020.563221:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134347008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697020.563233:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697020.563236:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697020.563237:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697020.563239:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4922 00000100:00080000:1.0:1787697020.563242:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697020.563245:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697020.563249:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697020.563258:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134347136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697020.563264:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697020.563266:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697020.563268:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697020.563271:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4922 00000100:00080000:1.0:1787697020.563273:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697020.563276:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697020.563279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697020.563296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134347264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697020.563302:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697020.563305:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697020.563308:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697020.563309:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697020.563313:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4922 00000100:00080000:1.0:1787697020.563317:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697020.563320:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697020.563325:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697020.563334:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134347392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697020.563345:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697020.563348:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697020.563349:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697020.563351:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4922 00000100:00080000:1.0:1787697020.563353:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697020.563355:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697020.563358:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697020.563371:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697020/real 1787697020] req@ffff9d85b9d2dc00 x1874531134346752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.563372:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134347520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697020.563412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134346752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697020.563422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697020.563427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697020.563432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134346752:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697020.563437:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134346880:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697020.563454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697020.563463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697020/real 1787697020] req@ffff9d85b9d2e680 x1874531134346880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697020.563471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134346880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697020.563477:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697020.563479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697020.563482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134346880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697020.563505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697020.563517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697020.563523:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697020/real 1787697020] req@ffff9d85b9d2c700 x1874531134347008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697020.563531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134347008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697020.563537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697020.563538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697020.563540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697020.563544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697020.563554:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697020.563561:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697020/real 1787697020] req@ffff9d85b9d2ed80 x1874531134347136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697020.563568:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134347136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697020.563574:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697020.563576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697020.563577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697020.563581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134347264:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697020.563592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697020.563598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697020/real 1787697020] req@ffff9d85b9d2d500 x1874531134347264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697020.563605:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134347264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697020.563610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697020.563612:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697020.563614:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134347264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697020.563617:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134347392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697020.563627:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697020.563639:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697020/real 1787697020] req@ffff9d85b9d2f800 x1874531134347392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697020.563650:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134347392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697020.563656:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697020.563658:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697020.563660:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134347392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697020.563663:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134347520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697020.563674:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697020.563680:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697020/real 1787697020] req@ffff9d85b9d2c380 x1874531134347520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697020.563687:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134347520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697020.563693:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697020.563695:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697020.563696:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134347520:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697021.139035:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697022.163059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697023.187073:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697024.211093:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697025.235032:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697025.682980:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697025.682988:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697025.682992:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697025.683001:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4927 00000100:00080000:1.0:1787697025.683011:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697025.683017:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697025.683051:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134347648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697025.683074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697025.683077:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697025.683079:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697025.683082:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4927 00000100:00080000:1.0:1787697025.683085:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697025.683088:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697025.683097:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697025.683107:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134347776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697025.683115:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697025.683118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697025.683119:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697025.683122:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4927 00000100:00080000:1.0:1787697025.683124:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697025.683126:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697025.683130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697025.683138:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134347904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697025.683152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697025.683152:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134347648:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697025.683161:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697025.683166:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697025.683174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4927 00000100:00080000:1.0:1787697025.683176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697025.683179:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697025.683181:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697025.683190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134348032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697025.683196:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697025.683198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697025.683200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697025.683203:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4927 00000100:00080000:1.0:1787697025.683205:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697025.683208:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697025.683212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697025.683219:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134348160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697025.683225:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697025.683227:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697025.683228:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697025.683231:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4927 00000100:00080000:1.0:1787697025.683234:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697025.683236:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697025.683241:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697025.683247:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134348288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697025.683253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697025.683255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697025.683256:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697025.683260:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4927 00000100:00080000:1.0:1787697025.683262:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697025.683264:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697025.683268:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697025.683277:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134348416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697025.683277:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697025.683347:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697025/real 1787697025] req@ffff9d85b9d2ca80 x1874531134347648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697025.683372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134347648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697025.683398:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697025.683404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697025.683410:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134347648:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697025.683419:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347776:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697025.683449:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697025.683465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697025/real 1787697025] req@ffff9d85b9d2f480 x1874531134347776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697025.683479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134347776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697025.683489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697025.683493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697025.683497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697025.683528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697025.683549:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697025.683561:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697025/real 1787697025] req@ffff9d85b9d2c000 x1874531134347904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697025.683574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134347904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697025.683583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697025.683587:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697025.683591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134347904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697025.683597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697025.683615:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697025.683624:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697025/real 1787697025] req@ffff9d85b9d2ce00 x1874531134348032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697025.683632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134348032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697025.683639:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697025.683641:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697025.683643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697025.683647:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134348160:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697025.683659:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697025.683666:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697025/real 1787697025] req@ffff9d85b9d2e300 x1874531134348160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697025.683673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134348160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697025.683678:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697025.683680:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697025.683682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134348160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697025.683685:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134348288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697025.683696:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697025.683701:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697025/real 1787697025] req@ffff9d85b9d2fb80 x1874531134348288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697025.683709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134348288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697025.683714:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697025.683717:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697025.683719:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134348288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697025.683722:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134348416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697025.683732:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697025.683737:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697025/real 1787697025] req@ffff9d85b9d2c380 x1874531134348416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697025.683762:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134348416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697025.683768:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697025.683770:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697025.683772:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134348416:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697026.259046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697027.283058:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697028.307043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697029.331021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697030.355013:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697030.802997:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697030.803008:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697030.803012:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697030.803024:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4932 00000100:00080000:1.0:1787697030.803036:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697030.803044:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697030.803088:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134348544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697030.803120:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697030.803125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697030.803127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697030.803132:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4932 00000100:00080000:1.0:1787697030.803137:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697030.803141:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697030.803153:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697030.803169:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134348544:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697030.803174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134348672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697030.803199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697030.803203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697030.803206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697030.803211:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4932 00000100:00080000:1.0:1787697030.803218:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697030.803224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:3.0:1787697030.803227:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697030.803234:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697030.803252:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134348800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697030.803263:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697030.803267:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697030.803270:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697030.803274:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697030/real 1787697030] req@ffff9d85b9d2f800 x1874531134348544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697030.803278:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4932 00000100:00080000:1.0:1787697030.803287:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697030.803294:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697030.803299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134348544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697030.803301:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697030.803313:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697030.803317:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697030.803323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134348544:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697030.803324:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134348928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697030.803338:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697030.803342:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787697030.803345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348672:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697030.803347:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697030.803356:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4932 00000100:00080000:1.0:1787697030.803361:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697030.803367:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:3.0:1787697030.803369:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697030.803377:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697030.803382:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697030/real 1787697030] req@ffff9d85b9d2d500 x1874531134348672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697030.803394:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134349056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697030.803395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134348672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697030.803422:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697030.803428:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697030.803430:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697030.803439:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4932 00000100:00080000:3.0:1787697030.803440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697030.803444:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697030.803449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697030.803454:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697030.803458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348672:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787697030.803463:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697030.803469:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697030.803480:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134349184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697030.803492:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697030.803498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697030.803499:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697030.803503:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697030.803510:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4932 00000100:00080000:1.0:1787697030.803515:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697030.803516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697030/real 1787697030] req@ffff9d85b9d2ed80 x1874531134348800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697030.803523:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697030.803534:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697030.803541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134348800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697030.803550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697030.803552:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134349312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697030.803557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697030.803566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697030.803579:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697030.803602:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697030.803613:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697030/real 1787697030] req@ffff9d85b9d2c700 x1874531134348928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697030.803626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134348928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697030.803634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697030.803637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697030.803641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134348928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697030.803692:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349056:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697030.803712:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697030.803723:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697030/real 1787697030] req@ffff9d85b9d2e680 x1874531134349056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697030.803734:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134349056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697030.803742:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697030.803745:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697030.803748:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697030.803753:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697030.803769:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697030.803779:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697030/real 1787697030] req@ffff9d85b9d2dc00 x1874531134349184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697030.803790:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134349184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697030.803805:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697030.803808:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697030.803812:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697030.803818:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697030.803837:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697030.803847:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697030/real 1787697030] req@ffff9d85b9d2f800 x1874531134349312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697030.803886:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134349312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697030.803894:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697030.803897:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697030.803900:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349312:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697031.379021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697032.403096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697033.427030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697034.451033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697035.475027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697035.922998:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697035.923012:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697035.923018:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697035.923039:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4937 00000100:00080000:0.0:1787697035.923055:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697035.923062:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697035.923109:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134349440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697035.923134:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697035.923136:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697035.923138:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697035.923141:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4937 00000100:00080000:0.0:1787697035.923143:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697035.923145:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697035.923153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697035.923162:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134349568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697035.923169:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697035.923171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697035.923172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697035.923177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4937 00000100:00100000:3.0:1787697035.923178:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134349440:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697035.923179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697035.923186:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697035.923191:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697035.923198:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134349696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697035.923203:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697035.923206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697035.923207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697035.923209:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4937 00000100:00080000:0.0:1787697035.923211:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697035.923213:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697035.923215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697035.923223:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134349824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697035.923228:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697035.923229:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697035.923230:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697035.923233:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4937 00000100:00080000:0.0:1787697035.923235:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697035.923238:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697035.923240:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697035.923245:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134349952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697035.923251:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697035.923253:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697035.923255:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697035.923258:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4937 00000100:00080000:0.0:1787697035.923262:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697035.923266:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697035.923269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697035.923276:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697035.923289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134350080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697035.923299:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697035.923302:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697035.923304:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697035.923309:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4937 00000100:00080000:0.0:1787697035.923313:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697035.923316:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697035.923320:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697035/real 1787697035] req@ffff9d85b9d0a680 x1874531134349440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697035.923322:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697035.923336:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134350208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697035.923340:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134349440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697035.923355:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697035.923359:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697035.923362:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134349440:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697035.923384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134349568:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697035.923399:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697035.923407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697035/real 1787697035] req@ffff9d85b9d0ad80 x1874531134349568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697035.923415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134349568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697035.923420:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697035.923422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697035.923424:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134349568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697035.923427:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134349696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697035.923436:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697035.923442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697035/real 1787697035] req@ffff9d85b9d0bb80 x1874531134349696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697035.923448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134349696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697035.923453:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697035.923455:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697035.923456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134349696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697035.923459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134349824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697035.923468:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697035.923474:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697035/real 1787697035] req@ffff9d85b9d08000 x1874531134349824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697035.923480:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134349824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697035.923487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697035.923489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697035.923492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134349824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697035.923498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697035.923515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697035.923525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697035/real 1787697035] req@ffff9d85b9d08a80 x1874531134349952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697035.923536:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134349952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697035.923541:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697035.923543:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697035.923545:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134349952:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697035.923548:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697035.923561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697035.923601:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697035/real 1787697035] req@ffff9d85b9d08e00 x1874531134350080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697035.923614:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134350080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697035.923621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697035.923623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697035.923625:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697035.923629:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697035.923643:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697035.923649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697035/real 1787697035] req@ffff9d85b9d0b480 x1874531134350208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697035.923656:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134350208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697035.923661:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697035.923670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697035.923672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350208:192.168.201.137@tcp:8:kworker.0 00010000:00010000:0.0:1787697036.498998:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697037.523031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697038.546949:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697039.571009:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697040.595003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697041.042934:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697041.042944:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697041.042949:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697041.042959:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4942 00000100:00080000:0.0:1787697041.042971:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697041.042979:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697041.043028:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134350336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697041.043055:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697041.043058:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697041.043059:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697041.043062:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4942 00000100:00080000:0.0:1787697041.043064:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697041.043066:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697041.043075:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697041.043083:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134350464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697041.043090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697041.043093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697041.043095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697041.043098:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4942 00000100:00080000:0.0:1787697041.043101:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697041.043103:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697041.043108:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697041.043115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134350592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697041.043121:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697041.043123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697041.043125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697041.043127:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4942 00000100:00080000:0.0:1787697041.043130:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697041.043139:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697041.043146:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697041.043148:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134350336:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697041.043159:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134350720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697041.043167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697041.043171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697041.043173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697041.043179:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4942 00000100:00080000:0.0:1787697041.043183:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697041.043188:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697041.043194:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697041.043209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134350848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697041.043219:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697041.043222:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697041.043229:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697041.043229:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697041.043240:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4942 00000100:00080000:0.0:1787697041.043246:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697041.043251:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697041.043256:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697041.043266:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134350976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697041.043277:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697041.043281:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697041/real 1787697041] req@ffff9d85b9d0a300 x1874531134350336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697041.043282:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697041.043286:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697041.043298:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4942 00000100:00080000:0.0:1787697041.043308:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697041.043314:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697041.043320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134350336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697041.043321:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697041.043341:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134351104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697041.043342:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697041.043358:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697041.043375:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134350336:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697041.043386:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134350464:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697041.043404:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697041.043412:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697041/real 1787697041] req@ffff9d85b9d09180 x1874531134350464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697041.043420:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134350464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697041.043425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697041.043428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697041.043430:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134350464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697041.043433:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134350592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697041.043444:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697041.043450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697041/real 1787697041] req@ffff9d85b9d0b800 x1874531134350592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697041.043457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134350592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697041.043462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697041.043464:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697041.043465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134350592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697041.043502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134350720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697041.043518:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697041.043526:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697041/real 1787697041] req@ffff9d85b9d08700 x1874531134350720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697041.043533:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134350720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697041.043539:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697041.043541:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697041.043543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134350720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697041.043547:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350848:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697041.043557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697041.043563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697041/real 1787697041] req@ffff9d85b9d09f80 x1874531134350848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697041.043570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134350848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697041.043575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697041.043577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697041.043578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350848:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697041.043582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697041.043591:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697041.043596:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697041/real 1787697041] req@ffff9d85b9d09c00 x1874531134350976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697041.043603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134350976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697041.043609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697041.043611:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697041.043613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134350976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697041.043616:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134351104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697041.043626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697041.043631:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697041/real 1787697041] req@ffff9d85b9d09500 x1874531134351104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697041.043637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134351104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697041.043642:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697041.043645:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697041.043647:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134351104:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697041.619102:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697042.643130:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697043.667063:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697044.691093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697045.715025:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697046.162949:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697046.162957:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697046.162960:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697046.162968:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4947 00000100:00080000:3.0:1787697046.162978:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697046.162984:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697046.163021:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134351232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697046.163050:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697046.163054:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697046.163056:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697046.163060:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4947 00000100:00080000:3.0:1787697046.163064:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697046.163069:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697046.163078:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697046.163086:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134351232:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697046.163093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134351360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697046.163101:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697046.163103:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697046.163104:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697046.163106:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4947 00000100:00080000:3.0:1787697046.163109:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697046.163111:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697046.163115:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697046.163123:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134351488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697046.163129:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697046.163131:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697046.163132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697046.163134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4947 00000100:00080000:3.0:1787697046.163137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697046.163139:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697046.163142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697046.163149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134351616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697046.163155:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697046.163157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697046.163158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697046.163160:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4947 00000100:00080000:3.0:1787697046.163162:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697046.163165:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697046.163168:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697046.163175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134351744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697046.163179:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134351360:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697046.163185:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697046.163187:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697046.163188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697046.163191:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4947 00000100:00080000:3.0:1787697046.163193:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697046.163196:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697046.163199:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697046.163206:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134351872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697046.163215:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134351488:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787697046.163217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697046.163223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697046.163224:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697046.163226:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4947 00000100:00080000:3.0:1787697046.163229:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697046.163231:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697046.163234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697046.163241:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134352000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697046.163258:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134351616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697046.163288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134351744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697046.163309:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134351872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697046.163327:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352000:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697046.739011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697047.763021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697048.786972:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787697049.235086:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787697049.235122:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697049.235139:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697049.235183:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697049.235192:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697049.235199:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697049.235206:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697049.235213:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697049.235220:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787697049.235306:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697046/real 1787697049] req@ffff9d85c0fb2300 x1874531134351744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697049.235331:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134351744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697049.235375:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697049.235380:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697049.235385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134351744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697049.235392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697046/real 1787697049] req@ffff9d85c0fb1880 x1874531134351872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697049.235405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134351872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697049.235419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697049.235423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697049.235426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134351872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697049.235433:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697046/real 1787697049] req@ffff9d85c0fb2a00 x1874531134352000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697049.235445:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134352000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697049.235454:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697049.235457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697049.235460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697049.235465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697046/real 1787697049] req@ffff9d85c0fb2680 x1874531134351360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697049.235474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134351360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697049.235481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697049.235484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697049.235487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134351360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697049.235492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697046/real 1787697049] req@ffff9d85c0fb3100 x1874531134351488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697049.235502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134351488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697049.235509:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697049.235511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697049.235514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134351488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697049.235519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697046/real 1787697049] req@ffff9d85c0fb3480 x1874531134351616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697049.235528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134351616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697049.235535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697049.235538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697049.235540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134351616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697049.235545:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697046/real 1787697049] req@ffff9d85c0fb1500 x1874531134351232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697049.235555:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134351232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697049.235561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697049.235564:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697049.235567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134351232:192.168.201.137@tcp:250:kworker.0 00010000:00010000:3.0:1787697049.811022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697050.834973:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697051.282922:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697051.282932:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697051.282936:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697051.282945:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4953 00000100:00080000:3.0:1787697051.282954:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697051.282961:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697051.283001:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134352128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697051.283023:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697051.283026:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697051.283028:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697051.283033:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4953 00000100:00080000:3.0:1787697051.283037:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697051.283042:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697051.283052:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697051.283056:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134352128:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697051.283068:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134352256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697051.283079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697051.283082:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697051.283084:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697051.283087:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4953 00000100:00080000:3.0:1787697051.283091:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697051.283095:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697051.283101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697051.283126:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697051.283128:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134352384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697051.283137:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697051.283140:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697051.283141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697051.283144:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4953 00000100:00080000:3.0:1787697051.283147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697051.283149:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697051.283153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697051.283163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134352512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697051.283177:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697051/real 1787697051] req@ffff9d85c0fb0700 x1874531134352128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697051.283178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697051.283187:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697051.283191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697051.283198:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4953 00000100:00080000:3.0:1787697051.283202:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697051.283207:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134352128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697051.283217:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697051.283221:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697051.283224:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697051.283229:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697051.283235:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134352128:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697051.283240:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134352640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697051.283250:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697051.283254:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697051.283255:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697051.283259:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4953 00000100:00080000:3.0:1787697051.283262:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697051.283263:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134352256:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697051.283269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697051.283276:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697051.283286:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134352768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697051.283296:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697051.283315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697051.283318:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697051.283321:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4953 00000100:00080000:3.0:1787697051.283326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697051.283329:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787697051.283334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697051.283335:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697051.283348:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134352896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697051.283352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697051/real 1787697051] req@ffff9d85c0fb0000 x1874531134352256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697051.283386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134352256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697051.283395:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697051.283398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697051.283401:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134352256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697051.283407:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134352384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697051.283428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697051.283439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697051/real 1787697051] req@ffff9d85c0fb1f80 x1874531134352384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697051.283449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134352384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697051.283456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697051.283459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697051.283461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134352384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697051.283466:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134352512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697051.283488:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697051.283498:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697051/real 1787697051] req@ffff9d85c0fb1500 x1874531134352512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697051.283513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134352512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697051.283520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697051.283522:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697051.283525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134352512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697051.283529:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352640:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697051.283552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697051.283563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697051/real 1787697051] req@ffff9d85c0fb3480 x1874531134352640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697051.283573:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134352640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697051.283580:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697051.283583:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697051.283585:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697051.283601:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697051.283619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697051.283628:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697051/real 1787697051] req@ffff9d85c0fb3100 x1874531134352768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697051.283637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134352768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697051.283648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697051.283651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697051.283654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697051.283658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697051.283675:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697051.283684:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697051/real 1787697051] req@ffff9d85c0fb2680 x1874531134352896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697051.283693:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134352896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697051.283700:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697051.283703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697051.283705:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134352896:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697051.859014:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697052.882957:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697053.907015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697054.931002:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697055.955020:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697056.402936:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697056.402944:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697056.402948:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697056.402955:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4958 00000100:00080000:3.0:1787697056.402964:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697056.402969:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697056.402999:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134353024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697056.403019:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697056.403022:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697056.403023:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697056.403026:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4958 00000100:00080000:3.0:1787697056.403028:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697056.403030:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697056.403036:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697056.403043:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134353152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697056.403054:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697056.403057:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697056.403059:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697056.403063:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4958 00000100:00080000:3.0:1787697056.403066:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697056.403071:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134353024:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697056.403072:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697056.403081:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697056.403094:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134353280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697056.403104:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697056.403107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697056.403108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697056.403110:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4958 00000100:00080000:3.0:1787697056.403113:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697056.403116:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697056.403119:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697056.403127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134353408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697056.403133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697056.403137:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697056.403137:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697056.403141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697056.403145:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4958 00000100:00080000:3.0:1787697056.403148:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697056.403150:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697056.403153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697056.403160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134353536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697056.403164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697056.403166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697056.403167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697056.403169:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4958 00000100:00080000:3.0:1787697056.403172:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697056.403174:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697056.403176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697056.403182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134353664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697056.403187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697056.403189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697056.403192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697056.403193:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697056/real 1787697056] req@ffff9d85c0fb2a00 x1874531134353024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697056.403196:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4958 00000100:00080000:3.0:1787697056.403200:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697056.403206:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697056.403228:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697056.403232:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134353024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697056.403243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134353792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697056.403245:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697056.403255:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697056.403262:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134353024:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697056.403273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134353152:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697056.403296:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697056.403308:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697056/real 1787697056] req@ffff9d85c0fb1880 x1874531134353152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697056.403320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134353152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697056.403327:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697056.403331:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697056.403335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134353152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697056.403354:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134353280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697056.403373:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697056.403384:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697056/real 1787697056] req@ffff9d85c0fb2300 x1874531134353280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697056.403395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134353280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697056.403403:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697056.403406:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697056.403409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134353280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697056.403415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134353408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697056.403433:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697056.403444:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697056/real 1787697056] req@ffff9d85c0fb0a80 x1874531134353408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697056.403455:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134353408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697056.403463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697056.403465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697056.403468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134353408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697056.403474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134353536:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697056.403492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697056.403502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697056/real 1787697056] req@ffff9d85c0fb0e00 x1874531134353536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697056.403512:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134353536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697056.403519:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697056.403522:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697056.403525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134353536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697056.403531:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134353664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697056.403546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697056.403555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697056/real 1787697056] req@ffff9d85c0fb3800 x1874531134353664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697056.403566:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134353664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697056.403579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697056.403582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697056.403585:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134353664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697056.403599:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134353792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697056.403616:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697056.403625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697056/real 1787697056] req@ffff9d85c0fb1c00 x1874531134353792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697056.403635:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134353792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697056.403642:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697056.403645:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697056.403648:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134353792:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697056.979007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697058.003008:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697059.027051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697060.051033:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697061.075030:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697061.522953:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697061.522965:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697061.522969:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697061.522979:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4963 00000100:00080000:3.0:1787697061.522989:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697061.522996:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697061.523030:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134353920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697061.523057:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697061.523060:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697061.523062:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697061.523066:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4963 00000100:00080000:3.0:1787697061.523069:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697061.523073:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697061.523081:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697061.523093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134354048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697061.523104:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134353920:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697061.523107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697061.523114:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697061.523115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697061.523120:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4963 00000100:00080000:3.0:1787697061.523149:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697061.523154:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697061.523159:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697061.523171:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134354176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697061.523181:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697061.523182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697061.523187:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697061.523190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697061.523201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4963 00000100:00080000:3.0:1787697061.523205:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697061.523209:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697061.523214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697061.523224:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697061/real 1787697061] req@ffff9d85c0fb3b80 x1874531134353920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697061.523229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134354304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697061.523247:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697061.523252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134353920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697061.523253:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697061.523257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697061.523266:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4963 00000100:00080000:3.0:1787697061.523271:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697061.523272:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697061.523277:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697061.523282:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787697061.523284:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697061.523288:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134353920:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697061.523298:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134354432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697061.523315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354048:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697061.523316:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697061.523324:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697061.523326:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697061.523331:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4963 00000100:00080000:3.0:1787697061.523336:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697061.523340:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787697061.523345:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697061.523348:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697061.523360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697061/real 1787697061] req@ffff9d85c0fb0380 x1874531134354048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697061.523370:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134354560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697061.523377:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134354048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697061.523388:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697061.523391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697061.523393:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697061.523397:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697061.523398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697061.523407:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4963 00000100:00100000:1.0:1787697061.523408:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354048:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697061.523413:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697061.523418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354176:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787697061.523420:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697061.523428:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697061.523440:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134354688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697061.523446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697061.523462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697061/real 1787697061] req@ffff9d85c0fb2d80 x1874531134354176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697061.523475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134354176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697061.523482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697061.523486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697061.523489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697061.523495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697061.523511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697061.523520:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697061/real 1787697061] req@ffff9d85c0fb1c00 x1874531134354304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697061.523531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134354304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697061.523538:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697061.523541:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697061.523543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697061.523556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134354432:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697061.523572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697061.523582:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697061/real 1787697061] req@ffff9d85c0fb3800 x1874531134354432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697061.523593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134354432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697061.523599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697061.523602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697061.523605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134354432:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697061.523610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134354560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697061.523626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697061.523636:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697061/real 1787697061] req@ffff9d85c0fb0e00 x1874531134354560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697061.523646:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134354560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697061.523659:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697061.523662:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697061.523665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134354560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697061.523670:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134354688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697061.523687:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697061.523698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697061/real 1787697061] req@ffff9d85c0fb0a80 x1874531134354688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697061.523708:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134354688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697061.523715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697061.523717:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697061.523721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134354688:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697062.098967:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697063.123034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697064.146994:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697065.170993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697066.195084:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697066.642945:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697066.642963:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697066.642969:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697066.642982:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4968 00000100:00080000:3.0:1787697066.642996:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697066.643006:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697066.643074:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134354816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697066.643110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697066.643115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697066.643116:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697066.643121:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4968 00000100:00080000:3.0:1787697066.643125:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697066.643129:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697066.643142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697066.643156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134354944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697066.643175:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134354816:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697066.643177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697066.643185:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697066.643187:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697066.643192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4968 00000100:00080000:3.0:1787697066.643195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697066.643199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697066.643205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697066.643218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134355072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697066.643228:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697066.643232:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697066.643234:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697066.643238:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4968 00000100:00080000:3.0:1787697066.643241:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697066.643245:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697066.643250:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697066.643262:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134355200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697066.643269:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697066.643276:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697066.643280:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697066.643282:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697066.643287:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4968 00000100:00080000:3.0:1787697066.643290:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697066.643296:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697066.643301:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697066.643312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134355328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697066.643321:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697066.643326:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697066.643327:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697066/real 1787697066] req@ffff9d85c0fb2300 x1874531134354816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697066.643329:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697066.643337:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4968 00000100:00080000:3.0:1787697066.643343:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697066.643348:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697066.643352:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697066.643364:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134355456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697066.643370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134354816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697066.643377:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697066.643388:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697066.643390:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697066.643392:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697066.643395:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4968 00000100:00080000:1.0:1787697066.643400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697066.643402:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697066.643407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134354816:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697066.643410:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697066.643429:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697066.643439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697066.643445:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134355584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697066.643468:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697066.643481:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697066/real 1787697066] req@ffff9d85c0fb1880 x1874531134354944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697066.643494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134354944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697066.643502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697066.643505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697066.643508:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134354944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697066.643514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697066.643532:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697066.643551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697066/real 1787697066] req@ffff9d85c0fb2a00 x1874531134355072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697066.643563:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134355072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697066.643571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697066.643574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697066.643577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697066.643583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697066.643601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697066.643611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697066/real 1787697066] req@ffff9d85c0fb2680 x1874531134355200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697066.643622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134355200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697066.643629:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697066.643645:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697066.643649:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697066.643655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134355328:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697066.643677:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697066.643689:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697066/real 1787697066] req@ffff9d85c0fb3100 x1874531134355328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697066.643702:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134355328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697066.643710:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697066.643713:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697066.643716:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134355328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697066.643721:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134355456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697066.643737:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697066.643747:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697066/real 1787697066] req@ffff9d85c0fb3480 x1874531134355456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697066.643757:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134355456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697066.643771:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697066.643774:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697066.643777:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134355456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697066.643794:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134355584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697066.643810:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697066.643819:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697066/real 1787697066] req@ffff9d85c0fb1500 x1874531134355584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697066.643831:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134355584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697066.643838:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697066.643842:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697066.643844:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134355584:192.168.201.137@tcp:8:kworker.0 00010000:00010000:0.0:1787697067.219035:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697068.243112:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697069.267013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697070.291031:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697071.315043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697071.762961:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697071.762971:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697071.762975:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697071.762985:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4973 00000100:00080000:3.0:1787697071.762995:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697071.763002:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697071.763032:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134355712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697071.763053:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697071.763056:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697071.763057:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697071.763059:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4973 00000100:00080000:3.0:1787697071.763062:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697071.763065:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697071.763072:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697071.763080:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134355840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697071.763087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697071.763091:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697071.763092:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134355712:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697071.763093:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697071.763100:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4973 00000100:00080000:3.0:1787697071.763102:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697071.763104:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697071.763107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697071.763114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134355968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697071.763120:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697071.763123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697071.763124:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697071.763126:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4973 00000100:00080000:3.0:1787697071.763128:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697071.763130:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697071.763133:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697071.763141:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134356096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697071.763149:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697071.763156:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697071.763158:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697071.763160:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697071.763163:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4973 00000100:00080000:3.0:1787697071.763165:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697071.763168:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697071.763170:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697071.763177:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134356224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697071.763182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697071.763184:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697071.763186:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697071.763188:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4973 00000100:00080000:3.0:1787697071.763190:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697071.763194:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697071/real 1787697071] req@ffff9d85c0fb1f80 x1874531134355712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697071.763194:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697071.763199:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697071.763209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134356352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697071.763215:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134355712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697071.763217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697071.763225:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697071.763228:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697071.763233:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697071.763233:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4973 00000100:00080000:3.0:1787697071.763239:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697071.763241:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697071.763244:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697071.763248:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134355712:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697071.763250:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697071.763259:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134356480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697071.763272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355840:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697071.763287:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697071.763294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697071/real 1787697071] req@ffff9d85c0fb0000 x1874531134355840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697071.763301:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134355840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697071.763307:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697071.763309:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697071.763311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697071.763314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697071.763323:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697071.763328:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697071/real 1787697071] req@ffff9d85c0fb0700 x1874531134355968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697071.763335:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134355968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697071.763339:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697071.763340:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697071.763342:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134355968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697071.763345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697071.763353:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697071.763358:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697071/real 1787697071] req@ffff9d85c0fb1500 x1874531134356096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697071.763365:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134356096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697071.763369:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697071.763371:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697071.763372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697071.763375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134356224:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697071.763384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697071.763388:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697071/real 1787697071] req@ffff9d85c0fb3480 x1874531134356224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697071.763394:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134356224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697071.763399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697071.763400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697071.763402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134356224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697071.763404:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134356352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697071.763412:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697071.763417:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697071/real 1787697071] req@ffff9d85c0fb3100 x1874531134356352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697071.763422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134356352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697071.763430:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697071.763432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697071.763433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134356352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697071.763436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134356480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697071.763444:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697071.763448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697071/real 1787697071] req@ffff9d85c0fb2680 x1874531134356480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697071.763454:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134356480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697071.763460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697071.763463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697071.763466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134356480:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697072.338969:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697073.363038:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697074.387035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697075.410993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697076.434991:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697076.882966:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697076.882976:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697076.882980:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697076.882989:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4978 00000100:00080000:3.0:1787697076.882998:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697076.883004:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697076.883039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134356608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697076.883068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697076.883072:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697076.883074:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697076.883077:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4978 00000100:00080000:3.0:1787697076.883081:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697076.883084:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697076.883093:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697076.883108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134356736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697076.883111:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134356608:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697076.883133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697076.883136:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697076.883137:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697076.883142:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4978 00000100:00080000:3.0:1787697076.883147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697076.883151:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697076.883156:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697076.883167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697076.883171:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134356864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697076.883182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697076.883186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697076.883188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697076.883192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4978 00000100:00080000:3.0:1787697076.883195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697076.883200:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697076.883206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697076.883207:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697076/real 1787697076] req@ffff9d85c0fb2a00 x1874531134356608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697076.883223:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134356992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697076.883226:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134356608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697076.883250:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697076.883252:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697076.883255:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697076.883260:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787697076.883261:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134356608:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697076.883264:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697076.883270:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4978 00000100:00080000:3.0:1787697076.883275:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697076.883279:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697076.883281:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356736:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787697076.883287:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697076.883299:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697076.883305:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134357120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697076.883309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697076/real 1787697076] req@ffff9d85c0fb1880 x1874531134356736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697076.883323:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697076.883328:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697076.883329:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697076.883332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134356736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697076.883336:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4978 00000100:00080000:3.0:1787697076.883342:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697076.883348:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697076.883349:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697076.883358:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787697076.883359:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697076.883363:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697076.883369:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697076.883375:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134357248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697076.883385:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697076.883388:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697076.883393:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697076.883395:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697076.883399:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4978 00000100:00080000:3.0:1787697076.883403:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697076.883408:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697076.883410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697076/real 1787697076] req@ffff9d85c0fb2300 x1874531134356864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697076.883414:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697076.883421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134356864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697076.883427:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697076.883430:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134357376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697076.883432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697076.883438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697076.883447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697076.883464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697076.883471:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697076/real 1787697076] req@ffff9d85c0fb0a80 x1874531134356992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697076.883479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134356992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697076.883484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697076.883485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697076.883487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134356992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697076.883497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134357120:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697076.883514:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697076.883523:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697076/real 1787697076] req@ffff9d85c0fb0e00 x1874531134357120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697076.883534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134357120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697076.883542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697076.883545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697076.883547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134357120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697076.883551:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134357248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697076.883563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697076.883569:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697076/real 1787697076] req@ffff9d85c0fb3800 x1874531134357248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697076.883575:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134357248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697076.883583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697076.883585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697076.883587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134357248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697076.883590:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134357376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697076.883597:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697076.883602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697076/real 1787697076] req@ffff9d85c0fb1c00 x1874531134357376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697076.883608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134357376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697076.883612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697076.883613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697076.883615:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134357376:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697077.459010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697078.483022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697079.506976:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697080.531020:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697081.554957:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697082.002963:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697082.002972:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697082.002976:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697082.002986:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4983 00000100:00080000:2.0:1787697082.002995:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697082.003001:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697082.003032:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134357504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697082.003056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697082.003059:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697082.003060:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697082.003063:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4983 00000100:00080000:2.0:1787697082.003065:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697082.003068:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697082.003076:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697082.003086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134357632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697082.003095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697082.003096:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134357504:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697082.003099:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697082.003102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697082.003107:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4983 00000100:00080000:2.0:1787697082.003108:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697082.003111:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697082.003114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697082.003123:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134357760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697082.003135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697082.003137:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697082.003139:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697082.003147:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4983 00000100:00080000:2.0:1787697082.003150:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697082.003152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697082.003155:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697082.003163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134357888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697082.003170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697082.003173:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697082.003177:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697082.003178:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697082.003181:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4983 00000100:00080000:2.0:1787697082.003183:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697082.003186:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697082.003190:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697082.003197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134358016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697082.003203:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697082.003205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697082.003207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697082.003209:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4983 00000100:00080000:2.0:1787697082.003212:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697082.003214:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697082.003217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697082.003225:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134358144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697082.003233:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697082/real 1787697082] req@ffff9d85babd6a00 x1874531134357504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697082.003233:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697082.003239:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697082.003241:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697082.003248:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4983 00000100:00080000:2.0:1787697082.003253:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697082.003259:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697082.003262:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697082.003273:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134357504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697082.003277:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134358272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697082.003295:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697082.003301:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697082.003305:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134357504:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697082.003326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134357632:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697082.003345:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697082.003355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697082/real 1787697082] req@ffff9d85babd6680 x1874531134357632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697082.003364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134357632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697082.003370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697082.003372:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697082.003374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134357632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697082.003378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134357760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697082.003390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697082.003396:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697082/real 1787697082] req@ffff9d85babd4000 x1874531134357760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697082.003404:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134357760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697082.003409:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697082.003411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697082.003413:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134357760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697082.003416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134357888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697082.003426:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697082.003432:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697082/real 1787697082] req@ffff9d85babd4380 x1874531134357888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697082.003440:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134357888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697082.003445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697082.003447:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697082.003449:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134357888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697082.003452:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697082.003461:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697082.003467:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697082/real 1787697082] req@ffff9d85babd6300 x1874531134358016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697082.003473:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134358016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697082.003478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697082.003480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697082.003482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697082.003485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697082.003495:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697082.003502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697082/real 1787697082] req@ffff9d85babd7480 x1874531134358144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697082.003514:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134358144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697082.003529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697082.003532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697082.003536:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697082.003542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697082.003563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697082.003575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697082/real 1787697082] req@ffff9d85babd5c00 x1874531134358272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697082.003587:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134358272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697082.003597:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697082.003601:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697082.003604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358272:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697082.579033:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697083.603034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697084.627008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697085.650999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697086.675027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697087.122974:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697087.122985:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697087.122990:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697087.123000:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4988 00000100:00080000:3.0:1787697087.123011:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697087.123018:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697087.123056:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134358400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697087.123087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697087.123096:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697087.123097:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697087.123101:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4988 00000100:00080000:3.0:1787697087.123105:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697087.123108:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697087.123116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697087.123128:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134358528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697087.123136:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697087.123140:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697087.123141:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134358400:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697087.123142:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697087.123147:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4988 00000100:00080000:3.0:1787697087.123152:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697087.123156:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697087.123159:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697087.123169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134358656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697087.123175:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697087.123178:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697087.123179:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697087.123182:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4988 00000100:00080000:3.0:1787697087.123184:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697087.123187:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697087.123191:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697087.123203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134358784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697087.123210:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697087.123218:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697087.123222:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697087.123225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697087.123230:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4988 00000100:00080000:3.0:1787697087.123236:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697087.123241:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697087.123247:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697087.123261:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697087/real 1787697087] req@ffff9d85c0fb2d80 x1874531134358400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697087.123263:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134358912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697087.123276:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697087.123281:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697087.123285:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697087.123291:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4988 00000100:00080000:3.0:1787697087.123294:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697087.123298:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697087.123302:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697087.123305:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134358400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697087.123312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134359040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697087.123328:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697087.123330:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697087.123331:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697087.123336:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4988 00000100:00080000:3.0:1787697087.123339:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697087.123340:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697087.123343:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697087.123348:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697087.123350:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697087.123356:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134359168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697087.123360:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134358400:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697087.123392:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134358528:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697087.123415:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697087.123425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697087/real 1787697087] req@ffff9d85c0fb0380 x1874531134358528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697087.123434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134358528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697087.123440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697087.123442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697087.123445:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134358528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697087.123448:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134358656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697087.123460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697087.123466:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697087/real 1787697087] req@ffff9d85c0fb3b80 x1874531134358656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697087.123473:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134358656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697087.123479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697087.123481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697087.123483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134358656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697087.123486:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134358784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697087.123501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697087.123511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697087/real 1787697087] req@ffff9d85c0fb1c00 x1874531134358784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697087.123524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134358784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697087.123533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697087.123536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697087.123539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134358784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697087.123545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358912:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697087.123563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697087.123571:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697087/real 1787697087] req@ffff9d85c0fb3800 x1874531134358912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697087.123579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134358912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697087.123584:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697087.123586:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697087.123588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134358912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697087.123591:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697087.123601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697087.123606:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697087/real 1787697087] req@ffff9d85c0fb0e00 x1874531134359040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697087.123613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134359040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697087.123623:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697087.123625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697087.123627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697087.123631:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697087.123639:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697087.123644:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697087/real 1787697087] req@ffff9d85c0fb0a80 x1874531134359168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697087.123651:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134359168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697087.123656:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697087.123658:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697087.123660:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359168:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697087.699008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697088.723009:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697089.747011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697090.771018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697091.794995:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697092.242942:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697092.242951:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697092.242954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697092.242963:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4994 00000100:00080000:1.0:1787697092.242972:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697092.242978:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697092.243010:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134359296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697092.243045:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697092.243050:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697092.243053:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697092.243059:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4994 00000100:00080000:1.0:1787697092.243063:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697092.243069:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697092.243078:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697092.243079:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134359296:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697092.243092:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134359424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697092.243101:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697092.243104:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697092.243105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697092.243108:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4994 00000100:00080000:1.0:1787697092.243110:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697092.243112:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697092.243116:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697092.243123:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134359552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697092.243130:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697092.243133:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697092.243134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697092.243136:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4994 00000100:00080000:1.0:1787697092.243139:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697092.243141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697092.243145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697092.243152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134359680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697092.243159:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697092.243164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697092.243168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697092.243169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697092.243172:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4994 00000100:00080000:1.0:1787697092.243175:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697092.243178:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697092.243181:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697092.243187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134359808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697092.243193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697092.243197:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697092.243199:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697092.243203:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4994 00000100:00080000:1.0:1787697092.243207:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697092.243212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697092.243217:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697092/real 1787697092] req@ffff9d85b9d2c380 x1874531134359296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697092.243220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697092.243240:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134359936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697092.243244:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134359296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697092.243252:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697092.243260:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697092.243264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697092.243272:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4994 00000100:00080000:1.0:1787697092.243277:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697092.243278:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697092.243282:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697092.243287:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:1.0:1787697092.243288:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697092.243294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134359296:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697092.243304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134360064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697092.243320:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134359424:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697092.243347:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697092.243359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697092/real 1787697092] req@ffff9d85b9d2fb80 x1874531134359424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697092.243372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134359424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697092.243382:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697092.243385:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697092.243389:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134359424:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697092.243395:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134359552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697092.243415:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697092.243426:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697092/real 1787697092] req@ffff9d85b9d2e300 x1874531134359552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697092.243439:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134359552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697092.243455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697092.243458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697092.243462:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134359552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697092.243468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134359680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697092.243485:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697092.243495:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697092/real 1787697092] req@ffff9d85b9d2ce00 x1874531134359680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697092.243507:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134359680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697092.243515:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697092.243519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697092.243522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134359680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697092.243527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359808:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697092.243542:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697092.243551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697092/real 1787697092] req@ffff9d85b9d2c000 x1874531134359808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697092.243563:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134359808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697092.243571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697092.243574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697092.243578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697092.243584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697092.243614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697092.243625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697092/real 1787697092] req@ffff9d85b9d2f480 x1874531134359936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697092.243637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134359936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697092.243646:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697092.243649:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697092.243653:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134359936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697092.243671:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697092.243687:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697092.243696:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697092/real 1787697092] req@ffff9d85b9d2ca80 x1874531134360064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697092.243708:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134360064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697092.243717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697092.243720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697092.243723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360064:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697092.819029:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697093.843022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697094.867024:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697095.890994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697096.914990:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697097.362945:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697097.362956:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697097.362960:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697097.362970:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 4999 00000100:00080000:1.0:1787697097.362982:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697097.362990:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697097.363029:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134360192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697097.363058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697097.363062:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697097.363064:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697097.363068:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4999 00000100:00080000:1.0:1787697097.363072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697097.363076:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697097.363085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697097.363101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134360320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697097.363111:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134360192:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697097.363129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697097.363133:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697097.363135:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697097.363140:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4999 00000100:00080000:1.0:1787697097.363144:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697097.363149:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697097.363155:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697097.363169:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134360448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697097.363180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697097.363181:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697097.363188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697097.363191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697097.363197:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 4999 00000100:00080000:1.0:1787697097.363202:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697097.363208:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697097.363213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697097.363228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134360576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697097.363233:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697097/real 1787697097] req@ffff9d85b9d2ca80 x1874531134360192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697097.363256:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697097.363260:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134360192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697097.363261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697097.363266:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697097.363280:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4999 00000100:00080000:3.0:1787697097.363281:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697097.363289:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697097.363290:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697097.363299:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697097.363301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134360192:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787697097.363309:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697097.363324:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134360704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697097.363332:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134360320:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697097.363338:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697097.363343:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697097.363345:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697097.363351:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4999 00000100:00080000:1.0:1787697097.363356:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697097.363361:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:3.0:1787697097.363366:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697097.363370:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697097.363383:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697097/real 1787697097] req@ffff9d85b9d2f480 x1874531134360320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697097.363386:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134360832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697097.363412:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697097.363417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134360320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697097.363418:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697097.363423:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697097.363431:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 4999 00000100:00080000:1.0:1787697097.363439:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697097.363440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697097.363446:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697097.363449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:1.0:1787697097.363455:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697097.363456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134360320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697097.363467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134360448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697097.363472:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134360960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697097.363493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697097.363525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697097/real 1787697097] req@ffff9d85b9d2c000 x1874531134360448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697097.363541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134360448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697097.363550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697097.363554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697097.363558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134360448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697097.363564:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134360576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697097.363593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697097.363605:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697097/real 1787697097] req@ffff9d85b9d2ce00 x1874531134360576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697097.363619:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134360576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697097.363628:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697097.363631:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697097.363635:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134360576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697097.363653:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360704:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697097.363676:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697097.363698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697097/real 1787697097] req@ffff9d85b9d2e300 x1874531134360704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697097.363711:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134360704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697097.363727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697097.363730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697097.363733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697097.363740:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697097.363758:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697097.363777:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697097/real 1787697097] req@ffff9d85b9d2fb80 x1874531134360832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697097.363789:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134360832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697097.363798:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697097.363801:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697097.363804:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697097.363810:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697097.363828:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697097.363839:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697097/real 1787697097] req@ffff9d85b9d2c380 x1874531134360960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697097.363886:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134360960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697097.363897:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697097.363900:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697097.363905:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134360960:192.168.201.137@tcp:8:kworker.0 00010000:00010000:0.0:1787697097.939035:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697098.962969:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697099.987065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697101.011005:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697102.035088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697102.483015:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697102.483027:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697102.483032:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697102.483044:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5004 00000100:00080000:3.0:1787697102.483057:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697102.483064:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697102.483116:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134361088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697102.483152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697102.483155:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697102.483157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697102.483161:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5004 00000100:00080000:3.0:1787697102.483164:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697102.483167:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697102.483178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697102.483189:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134361216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697102.483197:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697102.483200:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697102.483201:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697102.483204:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5004 00000100:00080000:3.0:1787697102.483207:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697102.483210:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697102.483215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697102.483224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134361344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697102.483232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697102.483236:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697102.483242:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697102.483245:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134361088:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697102.483249:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5004 00000100:00080000:3.0:1787697102.483259:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697102.483265:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697102.483269:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697102.483280:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134361472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697102.483287:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697102.483290:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697102.483292:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697102.483294:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5004 00000100:00080000:3.0:1787697102.483298:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697102.483301:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697102.483304:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697102.483311:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134361600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697102.483325:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697102.483325:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697102.483340:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697102.483343:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697102.483349:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5004 00000100:00080000:3.0:1787697102.483354:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697102.483358:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697102.483363:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697102.483374:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697102/real 1787697102] req@ffff9d85c0fb2300 x1874531134361088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697102.483375:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134361728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697102.483408:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697102.483435:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697102.483437:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697102.483442:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5004 00000100:00080000:3.0:1787697102.483447:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697102.483448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134361088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697102.483455:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697102.483468:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697102.483473:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697102.483479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697102.483484:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134361856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697102.483490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134361088:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697102.483515:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134361216:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697102.483541:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697102.483555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697102/real 1787697102] req@ffff9d85c0fb1880 x1874531134361216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697102.483570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134361216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697102.483576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697102.483579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697102.483581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134361216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697102.483601:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134361344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697102.483614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697102.483621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697102/real 1787697102] req@ffff9d85c0fb2a00 x1874531134361344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697102.483629:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134361344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697102.483634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697102.483635:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697102.483637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134361344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697102.483641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134361472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697102.483651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697102.483657:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697102/real 1787697102] req@ffff9d85c0fb2680 x1874531134361472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697102.483664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134361472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697102.483669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697102.483671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697102.483673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134361472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697102.483676:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134361600:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697102.483693:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697102.483700:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697102/real 1787697102] req@ffff9d85c0fb3100 x1874531134361600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697102.483707:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134361600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697102.483712:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697102.483714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697102.483716:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134361600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697102.483720:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134361728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697102.483731:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697102.483737:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697102/real 1787697102] req@ffff9d85c0fb3480 x1874531134361728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697102.483746:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134361728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697102.483754:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697102.483758:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697102.483761:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134361728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697102.483767:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134361856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697102.483786:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697102.483797:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697102/real 1787697102] req@ffff9d85c0fb1500 x1874531134361856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697102.483810:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134361856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697102.483815:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697102.483817:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697102.483820:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134361856:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697103.059054:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697104.083048:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697105.107040:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697106.131044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697107.155026:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697107.602953:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697107.602961:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697107.602966:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697107.602975:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5009 00000100:00080000:3.0:1787697107.602983:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697107.602989:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697107.603022:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134361984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697107.603044:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697107.603048:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697107.603050:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697107.603055:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5009 00000100:00080000:3.0:1787697107.603059:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697107.603063:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697107.603072:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697107.603086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134362112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697107.603087:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134361984:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697107.603097:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697107.603100:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697107.603102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697107.603105:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5009 00000100:00080000:3.0:1787697107.603108:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697107.603112:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697107.603117:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697107.603126:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134362240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697107.603140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697107.603143:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697107.603144:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697107.603148:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5009 00000800:00000100:1.0:1787697107.603149:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697107.603151:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697107.603154:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697107.603158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697107.603169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134362368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697107.603178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697107.603181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697107.603183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697107.603186:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5009 00000100:00080000:3.0:1787697107.603189:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697107.603193:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697107.603202:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697107.603205:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697107/real 1787697107] req@ffff9d85c0fb0700 x1874531134361984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697107.603212:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134362496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697107.603221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697107.603223:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134361984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697107.603225:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697107.603227:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697107.603230:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5009 00000100:00080000:1.0:1787697107.603234:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697107.603234:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697107.603238:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697107.603238:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697107.603242:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134361984:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697107.603243:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697107.603251:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134362624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697107.603260:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697107.603263:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697107.603265:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134362112:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697107.603265:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697107.603268:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5009 00000100:00080000:3.0:1787697107.603271:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697107.603275:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697107.603279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697107.603287:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697107.603288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134362752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697107.603299:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697107/real 1787697107] req@ffff9d85c0fb0000 x1874531134362112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697107.603310:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134362112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697107.603319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697107.603322:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697107.603326:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134362112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697107.603346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134362240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697107.603363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697107.603372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697107/real 1787697107] req@ffff9d85c0fb1f80 x1874531134362240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697107.603380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134362240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697107.603385:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697107.603387:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697107.603390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134362240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697107.603403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134362368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697107.603417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697107.603428:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697107/real 1787697107] req@ffff9d85c0fb1500 x1874531134362368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697107.603441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134362368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697107.603449:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697107.603453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697107.603456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134362368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697107.603462:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134362496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697107.603482:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697107.603492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697107/real 1787697107] req@ffff9d85c0fb3480 x1874531134362496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697107.603504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134362496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697107.603511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697107.603514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697107.603518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134362496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697107.603523:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134362624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697107.603538:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697107.603548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697107/real 1787697107] req@ffff9d85c0fb3100 x1874531134362624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697107.603558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134362624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697107.603565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697107.603568:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697107.603572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134362624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697107.603589:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134362752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697107.603605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697107.603615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697107/real 1787697107] req@ffff9d85c0fb2680 x1874531134362752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697107.603627:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134362752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697107.603635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697107.603638:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697107.603641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134362752:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697108.179011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697109.203046:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697110.227034:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697111.250961:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697112.274995:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697112.722951:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697112.722958:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697112.722961:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697112.722969:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5014 00000100:00080000:0.0:1787697112.722978:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697112.722983:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697112.723013:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134362880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697112.723032:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697112.723035:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697112.723036:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697112.723038:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5014 00000100:00080000:0.0:1787697112.723040:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697112.723042:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697112.723048:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697112.723060:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134363008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697112.723068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697112.723071:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697112.723072:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697112.723076:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5014 00000100:00100000:1.0:1787697112.723078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134362880:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697112.723079:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697112.723084:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697112.723090:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697112.723097:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134363136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697112.723103:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697112.723105:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697112.723105:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697112.723107:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5014 00000100:00080000:0.0:1787697112.723109:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697112.723111:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697112.723113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697112.723120:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134363264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697112.723125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697112.723127:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697112.723128:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697112.723130:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5014 00000100:00080000:0.0:1787697112.723137:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697112.723140:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697112.723143:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697112.723149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134363392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697112.723154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697112.723156:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697112.723157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697112.723162:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5014 00000100:00080000:0.0:1787697112.723164:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697112.723166:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697112.723169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697112.723176:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134363520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697112.723183:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697112.723186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697112.723186:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363008:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697112.723188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697112.723194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5014 00000100:00080000:0.0:1787697112.723196:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697112.723198:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697112.723200:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697112.723209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134363648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697112.723227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697112.723263:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697112.723282:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134363392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697112.723306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134363520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697112.723328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134363648:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697113.299025:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697114.323013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697115.347009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697115.795140:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697115.795183:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697115.795225:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697115.795264:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697115.795273:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697115.795281:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697115.795289:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697115.795297:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697115.795305:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787697115.795365:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697112/real 1787697115] req@ffff9d85b9d09180 x1874531134363520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697115.795394:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134363520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697115.795407:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697115.795411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697115.795415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134363520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697115.795424:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697112/real 1787697115] req@ffff9d85b9d0a300 x1874531134363648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697115.795436:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134363648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697115.795448:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697115.795452:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697115.795455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134363648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697115.795461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697112/real 1787697115] req@ffff9d85b9d09c00 x1874531134363008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697115.795472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134363008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697115.795478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697115.795482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697115.795485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697115.795491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697112/real 1787697115] req@ffff9d85b9d09f80 x1874531134363136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697115.795502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134363136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697115.795509:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697115.795513:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697115.795516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697115.795522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697112/real 1787697115] req@ffff9d85b9d08700 x1874531134363264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697115.795535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134363264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697115.795544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697115.795547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697115.795550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697115.795557:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697112/real 1787697115] req@ffff9d85b9d0b800 x1874531134363392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697115.795577:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134363392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697115.795584:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697115.795588:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697115.795591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134363392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697115.795597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697112/real 1787697115] req@ffff9d85b9d09500 x1874531134362880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697115.795610:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134362880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697115.795617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697115.795621:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697115.795624:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134362880:192.168.201.137@tcp:250:kworker.0 00010000:00010000:0.0:1787697116.371036:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697117.395118:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697117.842976:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697117.842987:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697117.842992:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697117.843001:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5019 00000100:00080000:3.0:1787697117.843011:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697117.843018:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697117.843049:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134363776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697117.843089:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697117.843093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697117.843095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697117.843099:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5019 00000100:00080000:3.0:1787697117.843102:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697117.843105:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697117.843116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697117.843131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134363904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697117.843140:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134363776:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697117.843143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697117.843148:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697117.843150:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697117.843164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5019 00000100:00080000:3.0:1787697117.843166:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697117.843169:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697117.843174:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697117.843186:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134364032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697117.843206:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697117.843210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697117.843212:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697117.843216:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5019 00000800:00000100:1.0:1787697117.843221:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697117.843223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697117.843229:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697117.843235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697117.843247:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134364160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697117.843255:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697117.843257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697117.843259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697117.843262:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5019 00000100:00080000:3.0:1787697117.843266:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697117.843271:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697117.843276:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697117/real 1787697117] req@ffff9d85c0fb2a00 x1874531134363776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697117.843278:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697117.843296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134364288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697117.843303:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134363776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697117.843310:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697117.843318:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697117.843320:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697117.843321:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697117.843327:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5019 00000100:00080000:1.0:1787697117.843328:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697117.843333:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697117.843335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134363776:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697117.843340:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697117.843348:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697117.843359:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134364416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697117.843367:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363904:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697117.843372:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697117.843377:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697117.843379:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697117.843384:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5019 00000100:00080000:3.0:1787697117.843388:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697117.843392:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697117.843397:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697117.843401:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697117.843413:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134364544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697117.843420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697117/real 1787697117] req@ffff9d85c0fb1880 x1874531134363904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697117.843434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134363904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697117.843443:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697117.843447:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697117.843450:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134363904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697117.843456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697117.843475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697117.843572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697117/real 1787697117] req@ffff9d85c0fb2300 x1874531134364032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697117.843590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134364032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697117.843604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697117.843607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697117.843610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697117.843617:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697117.843637:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697117.843648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697117/real 1787697117] req@ffff9d85c0fb0a80 x1874531134364160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697117.843661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134364160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697117.843669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697117.843672:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697117.843675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697117.843681:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134364288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697117.843700:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697117.843712:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697117/real 1787697117] req@ffff9d85c0fb0e00 x1874531134364288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697117.843722:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134364288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697117.843729:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697117.843732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697117.843736:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134364288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697117.843753:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134364416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697117.843772:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697117.843784:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697117/real 1787697117] req@ffff9d85c0fb3800 x1874531134364416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697117.843795:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134364416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697117.843804:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697117.843806:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697117.843809:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134364416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697117.843815:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134364544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697117.843834:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697117.843844:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697117/real 1787697117] req@ffff9d85c0fb1c00 x1874531134364544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697117.843893:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134364544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697117.843901:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697117.843904:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697117.843907:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134364544:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697118.419011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697119.442941:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697120.466956:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697121.491087:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697122.514990:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697122.962929:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697122.962938:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697122.962941:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697122.962950:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5024 00000100:00080000:3.0:1787697122.962960:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697122.962966:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697122.962999:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134364672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697122.963023:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697122.963026:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697122.963028:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697122.963032:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5024 00000100:00080000:3.0:1787697122.963035:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697122.963038:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697122.963046:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697122.963057:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134364800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697122.963080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697122.963083:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697122.963085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697122.963089:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134364672:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697122.963091:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5024 00000100:00080000:3.0:1787697122.963097:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697122.963102:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697122.963106:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697122.963117:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134364928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697122.963126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697122.963130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697122.963132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697122.963135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5024 00000100:00080000:3.0:1787697122.963139:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697122.963143:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697122.963146:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697122.963157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134365056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697122.963162:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697122.963178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697122.963181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697122.963184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697122.963188:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5024 00000100:00080000:3.0:1787697122.963192:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697122.963195:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697122.963199:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697122.963209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134365184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697122.963213:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697122/real 1787697122] req@ffff9d85c0fb3b80 x1874531134364672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697122.963225:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697122.963231:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697122.963233:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697122.963237:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5024 00000100:00100000:1.0:1787697122.963241:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134364672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697122.963242:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697122.963250:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697122.963257:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697122.963260:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697122.963267:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697122.963269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134365312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697122.963273:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134364672:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697122.963287:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697122.963289:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697122.963291:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697122.963295:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5024 00000100:00080000:3.0:1787697122.963299:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697122.963303:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364800:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697122.963306:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697122.963313:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697122.963324:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134365440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697122.963331:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697122.963349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697122/real 1787697122] req@ffff9d85c0fb0380 x1874531134364800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697122.963361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134364800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697122.963369:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697122.963372:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697122.963376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697122.963381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697122.963396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697122.963405:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697122/real 1787697122] req@ffff9d85c0fb2d80 x1874531134364928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697122.963415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134364928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697122.963428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697122.963430:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697122.963434:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134364928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697122.963447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697122.963462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697122.963470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697122/real 1787697122] req@ffff9d85c0fb1c00 x1874531134365056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697122.963481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134365056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697122.963488:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697122.963491:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697122.963494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697122.963499:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134365184:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697122.963521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697122.963530:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697122/real 1787697122] req@ffff9d85c0fb3800 x1874531134365184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697122.963540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134365184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697122.963548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697122.963551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697122.963554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134365184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697122.963559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134365312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697122.963573:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697122.963581:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697122/real 1787697122] req@ffff9d85c0fb0e00 x1874531134365312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697122.963590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134365312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697122.963598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697122.963600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697122.963603:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134365312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697122.963618:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134365440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697122.963632:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697122.963649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697122/real 1787697122] req@ffff9d85c0fb0a80 x1874531134365440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697122.963660:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134365440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697122.963667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697122.963670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697122.963672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134365440:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697123.539007:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697124.563015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697125.587048:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697126.610994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697127.634998:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697128.082958:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697128.082965:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697128.082969:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697128.083000:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5029 00000100:00080000:0.0:1787697128.083009:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697128.083015:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697128.083049:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134365568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697128.083075:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697128.083078:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697128.083080:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697128.083084:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5029 00000100:00080000:0.0:1787697128.083088:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697128.083093:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697128.083101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697128.083111:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134365568:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697128.083114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134365696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697128.083124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697128.083126:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697128.083127:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697128.083129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5029 00000100:00080000:0.0:1787697128.083132:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697128.083134:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697128.083137:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697128.083144:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134365824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697128.083154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697128.083157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697128.083158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697128.083160:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5029 00000100:00080000:0.0:1787697128.083161:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697128.083165:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697128.083168:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697128.083174:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697128.083175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134365952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697128.083182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697128.083185:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697128.083186:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697128.083189:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5029 00000100:00080000:0.0:1787697128.083191:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697128.083194:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697128.083197:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697128.083203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134366080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697128.083208:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697128.083210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697128.083211:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697128.083215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5029 00000100:00100000:1.0:1787697128.083216:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697128/real 1787697128] req@ffff9d85b9d0b480 x1874531134365568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697128.083219:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697128.083225:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697128.083231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697128.083239:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134365568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697128.083240:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134366208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697128.083263:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697128.083265:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697128.083268:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787697128.083272:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697128.083275:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134365568:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697128.083277:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697128.083285:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5029 00000100:00080000:0.0:1787697128.083289:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697128.083293:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697128.083297:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697128.083298:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697128.083312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134366336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697128.083320:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697128.083331:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697128/real 1787697128] req@ffff9d85b9d08e00 x1874531134365696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697128.083339:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134365696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697128.083345:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697128.083346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697128.083348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697128.083351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697128.083361:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697128.083366:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697128/real 1787697128] req@ffff9d85b9d08a80 x1874531134365824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697128.083372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134365824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697128.083380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697128.083383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697128.083385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697128.083387:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697128.083395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697128.083400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697128/real 1787697128] req@ffff9d85b9d08000 x1874531134365952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697128.083406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134365952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697128.083410:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697128.083411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697128.083413:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134365952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697128.083416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366080:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697128.083424:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697128.083429:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697128/real 1787697128] req@ffff9d85b9d0bb80 x1874531134366080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697128.083435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134366080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697128.083439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697128.083440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697128.083443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697128.083446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697128.083454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697128.083458:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697128/real 1787697128] req@ffff9d85b9d0ad80 x1874531134366208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697128.083464:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134366208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697128.083469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697128.083470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697128.083472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697128.083481:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697128.083490:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697128.083494:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697128/real 1787697128] req@ffff9d85b9d0a680 x1874531134366336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697128.083500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134366336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697128.083504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697128.083506:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697128.083507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366336:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697128.659010:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697129.683039:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697130.706964:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697131.731048:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697132.755048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697133.202923:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697133.202931:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697133.202934:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697133.202942:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5034 00000100:00080000:0.0:1787697133.202951:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697133.202958:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697133.202993:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134366464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697133.203014:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697133.203018:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697133.203020:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697133.203023:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5034 00000100:00080000:0.0:1787697133.203026:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697133.203029:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697133.203036:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697133.203046:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134366592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697133.203052:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697133.203054:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697133.203056:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697133.203065:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134366464:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697133.203066:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5034 00000100:00080000:0.0:1787697133.203073:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697133.203076:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697133.203080:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697133.203088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134366720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697133.203094:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697133.203097:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697133.203099:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697133.203102:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5034 00000100:00080000:0.0:1787697133.203105:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697133.203108:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697133.203114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697133.203125:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134366848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697133.203134:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697133.203137:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697133.203139:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697133.203142:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5034 00000100:00080000:0.0:1787697133.203145:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697133.203149:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697133.203153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697133.203159:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697133.203165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134366976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697133.203175:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697133.203179:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697133.203180:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697133.203184:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5034 00000100:00080000:0.0:1787697133.203187:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697133.203190:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697133.203194:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697133.203209:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134367104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697133.203210:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697133/real 1787697133] req@ffff9d85b9d0a680 x1874531134366464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697133.203215:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697133.203217:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697133.203218:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697133.203220:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5034 00000100:00080000:0.0:1787697133.203222:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697133.203224:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697133.203226:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697133.203235:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134367232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697133.203280:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134366464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697133.203291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697133.203295:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697133.203298:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134366464:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697133.203319:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134366592:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697133.203342:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697133.203353:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697133/real 1787697133] req@ffff9d85b9d0ad80 x1874531134366592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697133.203364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134366592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697133.203371:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697133.203374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697133.203376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134366592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697133.203382:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134366720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697133.203397:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697133.203406:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697133/real 1787697133] req@ffff9d85b9d0bb80 x1874531134366720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697133.203415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134366720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697133.203426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697133.203429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697133.203431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134366720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697133.203436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134366848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697133.203452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697133.203462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697133/real 1787697133] req@ffff9d85b9d08000 x1874531134366848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697133.203473:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134366848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697133.203480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697133.203483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697133.203486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134366848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697133.203492:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366976:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697133.203508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697133.203528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697133/real 1787697133] req@ffff9d85b9d08a80 x1874531134366976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697133.203539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134366976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697133.203546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697133.203549:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697133.203553:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134366976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697133.203558:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134367104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697133.203575:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697133.203585:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697133/real 1787697133] req@ffff9d85b9d08e00 x1874531134367104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697133.203627:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134367104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697133.203635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697133.203638:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697133.203642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134367104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697133.203647:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134367232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697133.203694:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697133.203704:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697133/real 1787697133] req@ffff9d85b9d0b480 x1874531134367232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697133.203732:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134367232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697133.203738:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697133.203740:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697133.203742:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134367232:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697133.778998:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697134.803028:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697135.826967:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697136.851025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697137.874956:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697138.322921:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697138.322930:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697138.322933:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697138.322941:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5040 00000100:00080000:0.0:1787697138.322949:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697138.322954:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697138.322982:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134367360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697138.323002:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697138.323004:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697138.323006:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697138.323008:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5040 00000100:00080000:0.0:1787697138.323010:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697138.323012:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697138.323019:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697138.323026:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134367488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697138.323036:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697138.323039:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697138.323040:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697138.323042:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5040 00000100:00080000:0.0:1787697138.323044:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697138.323049:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697138.323049:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134367360:192.168.201.137@tcp:250:kworker.0 00000008:00100000:0.0:1787697138.323053:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697138.323060:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134367616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697138.323065:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697138.323068:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697138.323069:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697138.323071:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5040 00000100:00080000:0.0:1787697138.323072:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697138.323074:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697138.323077:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697138.323088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134367744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697138.323097:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697138.323100:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697138.323102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697138.323105:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5040 00000100:00080000:0.0:1787697138.323110:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697138.323113:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787697138.323119:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787697138.323121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697138.323133:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134367872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697138.323139:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697138.323141:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697138.323143:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697138.323145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5040 00000100:00080000:0.0:1787697138.323148:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697138.323150:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697138.323153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697138.323166:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697138/real 1787697138] req@ffff9d85b9d09500 x1874531134367360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697138.323167:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134368000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697138.323193:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697138.323197:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697138.323199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697138.323202:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134367360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697138.323206:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5040 00000100:00080000:0.0:1787697138.323213:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697138.323218:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697138.323220:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787697138.323222:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697138.323228:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697138.323234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134368128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697138.323235:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134367360:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697138.323249:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134367488:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697138.323281:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697138.323293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697138/real 1787697138] req@ffff9d85b9d0b800 x1874531134367488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697138.323306:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134367488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697138.323314:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697138.323317:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697138.323320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134367488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697138.323340:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134367616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697138.323357:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697138.323369:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697138/real 1787697138] req@ffff9d85b9d08700 x1874531134367616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697138.323380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134367616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697138.323393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697138.323396:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697138.323407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134367616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697138.323413:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134367744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697138.323430:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697138.323441:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697138/real 1787697138] req@ffff9d85b9d09f80 x1874531134367744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697138.323452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134367744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697138.323466:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697138.323469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697138.323472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134367744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697138.323477:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134367872:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697138.323494:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697138.323504:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697138/real 1787697138] req@ffff9d85b9d09c00 x1874531134367872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697138.323516:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134367872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697138.323523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697138.323526:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697138.323530:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134367872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697138.323535:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697138.323550:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697138.323560:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697138/real 1787697138] req@ffff9d85b9d0a300 x1874531134368000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697138.323569:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134368000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697138.323577:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697138.323580:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697138.323583:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697138.323588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697138.323604:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697138.323613:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697138/real 1787697138] req@ffff9d85b9d09180 x1874531134368128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697138.323624:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134368128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697138.323631:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697138.323634:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697138.323637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368128:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697138.899012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697139.923018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697140.947058:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697141.970995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697142.995009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697143.442932:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697143.442943:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697143.442947:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697143.442957:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5045 00000100:00080000:1.0:1787697143.442968:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697143.442975:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697143.443033:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134368256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697143.443096:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134368256:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697143.443117:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697143.443123:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697143.443125:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697143.443131:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5045 00000100:00080000:1.0:1787697143.443143:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697143.443149:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:3.0:1787697143.443151:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697143.443180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697143.443195:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697143/real 1787697143] req@ffff9d85b9d2f800 x1874531134368256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697143.443198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134368384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697143.443221:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697143.443224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697143.443226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697143.443229:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5045 00000100:00080000:1.0:1787697143.443234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697143.443238:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134368256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697143.443241:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697143.443250:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697143.443255:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697143.443262:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697143.443264:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134368512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697143.443270:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134368256:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697143.443287:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697143.443291:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697143.443293:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697143.443295:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134368384:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697143.443300:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5045 00000100:00080000:1.0:1787697143.443304:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697143.443312:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697143.443317:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697143.443331:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697143.443332:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134368640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697143.443343:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697143/real 1787697143] req@ffff9d85b9d2dc00 x1874531134368384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697143.443346:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697143.443355:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697143.443358:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697143.443361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134368384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697143.443365:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5045 00000100:00080000:1.0:1787697143.443372:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697143.443374:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697143.443378:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697143.443379:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697143.443385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134368384:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787697143.443387:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697143.443390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134368512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697143.443405:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134368768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697143.443408:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697143.443417:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697143/real 1787697143] req@ffff9d85b9d2e680 x1874531134368512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697143.443419:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697143.443430:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697143.443432:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697143.443434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134368512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697143.443441:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5045 00000100:00080000:1.0:1787697143.443449:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697143.443450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697143.443455:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697143.443458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:1.0:1787697143.443462:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697143.443465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134368512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697143.443476:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134368896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697143.443486:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134368640:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787697143.443489:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697143.443494:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697143.443496:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697143.443501:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5045 00000100:00080000:1.0:1787697143.443506:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697143.443510:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697143.443514:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697143.443520:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697143.443521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697143/real 1787697143] req@ffff9d85b9d2c700 x1874531134368640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697143.443537:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134369024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697143.443538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134368640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697143.443554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697143.443556:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697143.443558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134368640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697143.443565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368768:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697143.443585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697143.443596:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697143/real 1787697143] req@ffff9d85b9d2ed80 x1874531134368768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697143.443608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134368768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697143.443615:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697143.443618:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697143.443621:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697143.443632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697143.443645:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697143.443652:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697143/real 1787697143] req@ffff9d85b9d2d500 x1874531134368896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697143.443658:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134368896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697143.443666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697143.443668:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697143.443670:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134368896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697143.443673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697143.443681:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697143.443686:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697143/real 1787697143] req@ffff9d85b9d2e680 x1874531134369024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697143.443691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134369024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697143.443696:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697143.443698:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697143.443699:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369024:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697144.019014:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697145.042992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697146.066996:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697147.090990:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697148.114948:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697148.562969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697148.562978:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697148.562981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697148.562990:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5050 00000100:00080000:2.0:1787697148.562999:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697148.563006:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697148.563039:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134369152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697148.563063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697148.563066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697148.563067:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697148.563070:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5050 00000100:00080000:2.0:1787697148.563073:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697148.563076:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697148.563086:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697148.563097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134369280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697148.563107:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697148.563111:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697148.563113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697148.563113:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134369152:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697148.563121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5050 00000100:00080000:2.0:1787697148.563125:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697148.563128:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697148.563132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697148.563144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134369408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697148.563152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697148.563156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697148.563158:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697148.563162:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5050 00000100:00080000:2.0:1787697148.563165:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697148.563169:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697148.563173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697148.563179:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697148.563185:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134369536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697148.563195:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697148.563198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697148.563200:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697148.563203:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5050 00000100:00080000:2.0:1787697148.563206:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697148.563210:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697148.563214:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697148.563222:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134369664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697148.563230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697148.563233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697148.563235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697148.563239:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5050 00000100:00100000:3.0:1787697148.563243:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697148/real 1787697148] req@ffff9d85babd7800 x1874531134369152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697148.563244:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697148.563252:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697148.563260:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697148.563273:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134369152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697148.563274:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134369792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697148.563303:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697148.563304:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697148.563306:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697148.563311:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697148.563315:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697148.563319:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5050 00000100:00100000:3.0:1787697148.563324:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134369152:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697148.563325:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697148.563333:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697148.563338:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697148.563348:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134369920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697148.563351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134369280:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697148.563381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697148.563406:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697148/real 1787697148] req@ffff9d85babd5500 x1874531134369280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697148.563418:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134369280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697148.563425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697148.563428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697148.563432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134369280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697148.563439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134369408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697148.563458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697148.563470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697148/real 1787697148] req@ffff9d85babd6d80 x1874531134369408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697148.563481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134369408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697148.563489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697148.563491:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697148.563495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134369408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697148.563501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134369536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697148.563519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697148.563529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697148/real 1787697148] req@ffff9d85babd5c00 x1874531134369536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697148.563540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134369536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697148.563547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697148.563551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697148.563554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134369536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697148.563559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369664:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697148.563577:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697148.563588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697148/real 1787697148] req@ffff9d85babd7480 x1874531134369664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697148.563598:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134369664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697148.563605:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697148.563607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697148.563610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697148.563615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697148.563664:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697148.563675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697148/real 1787697148] req@ffff9d85babd6300 x1874531134369792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697148.563691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134369792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697148.563699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697148.563701:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697148.563704:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697148.563719:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697148.563735:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697148.563743:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697148/real 1787697148] req@ffff9d85babd4380 x1874531134369920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697148.563753:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134369920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697148.563761:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697148.563764:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697148.563767:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134369920:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697149.139068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697150.163010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697151.187095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697152.211015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697153.235016:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697153.682973:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697153.682983:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697153.682987:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697153.682998:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5055 00000100:00080000:2.0:1787697153.683008:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697153.683016:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697153.683080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134370048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697153.683109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697153.683113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697153.683116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697153.683121:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5055 00000100:00080000:2.0:1787697153.683124:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697153.683128:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697153.683137:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697153.683151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134370176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697153.683158:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134370048:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697153.683164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697153.683168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697153.683170:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697153.683174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5055 00000100:00080000:2.0:1787697153.683179:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697153.683182:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697153.683188:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697153.683213:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134370304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697153.683223:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697153.683224:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697153.683230:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697153.683232:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697153.683237:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5055 00000100:00080000:2.0:1787697153.683248:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697153.683252:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697153.683258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697153.683269:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134370432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697153.683279:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697153.683282:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697153/real 1787697153] req@ffff9d85babd4000 x1874531134370048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697153.683283:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697153.683286:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697153.683295:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5055 00000100:00080000:2.0:1787697153.683302:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697153.683308:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697153.683312:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134370048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697153.683315:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697153.683330:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697153.683333:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134370560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697153.683337:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697153.683345:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134370048:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697153.683352:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697153.683356:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697153.683358:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697153.683363:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5055 00000100:00080000:2.0:1787697153.683367:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697153.683368:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134370176:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697153.683375:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697153.683381:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697153.683395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697153.683397:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134370688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697153.683410:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697153.683413:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697153/real 1787697153] req@ffff9d85babd6680 x1874531134370176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697153.683414:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697153.683418:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697153.683428:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5055 00000100:00080000:2.0:1787697153.683434:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697153.683439:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134370176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697153.683441:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697153.683449:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697153.683453:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.683458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697153.683463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134370176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697153.683471:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134370304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787697153.683475:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134370816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697153.683497:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697153.683513:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697153/real 1787697153] req@ffff9d85babd6a00 x1874531134370304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.683525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134370304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.683546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.683550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697153.683554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134370304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697153.683561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134370432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697153.683580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697153.683591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697153/real 1787697153] req@ffff9d85babd5180 x1874531134370432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.683603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134370432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.683610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.683613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697153.683617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134370432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697153.683622:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134370560:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697153.683642:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697153.683653:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697153/real 1787697153] req@ffff9d85babd4a80 x1874531134370560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.683664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134370560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.683671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.683674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697153.683677:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134370560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697153.683694:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134370688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697153.683714:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697153.683725:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697153/real 1787697153] req@ffff9d85babd7b80 x1874531134370688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.683736:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134370688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.683743:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.683746:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697153.683749:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134370688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697153.683754:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134370816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697153.683771:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697153.683780:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697153/real 1787697153] req@ffff9d85babd7100 x1874531134370816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.683791:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134370816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.683799:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.683802:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697153.683804:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134370816:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697154.259026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697155.282976:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697156.307017:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697157.330996:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697158.355025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697158.802992:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697158.802999:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697158.803002:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697158.803010:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5060 00000100:00080000:0.0:1787697158.803019:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697158.803024:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697158.803054:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134370944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697158.803075:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697158.803077:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697158.803079:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697158.803081:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5060 00000100:00080000:0.0:1787697158.803083:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697158.803085:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697158.803091:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697158.803099:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134371072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697158.803105:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697158.803107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697158.803114:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697158.803117:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5060 00000100:00080000:0.0:1787697158.803119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697158.803121:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697158.803126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697158.803131:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134370944:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697158.803135:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134371200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697158.803144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697158.803146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697158.803147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697158.803149:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5060 00000100:00080000:0.0:1787697158.803150:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697158.803152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697158.803154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697158.803161:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134371328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697158.803166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697158.803167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697158.803168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697158.803170:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5060 00000100:00080000:0.0:1787697158.803172:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697158.803175:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697158.803178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697158.803183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134371456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697158.803189:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697158.803191:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697158.803192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697158.803194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5060 00000100:00080000:0.0:1787697158.803196:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697158.803200:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697158.803202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697158.803206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697158.803212:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134371584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697158.803218:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697158.803220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697158.803221:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697158.803223:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5060 00000100:00080000:0.0:1787697158.803225:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697158.803227:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697158.803229:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697158.803236:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134371712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697158.803241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697158/real 1787697158] req@ffff9d85b9d09180 x1874531134370944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697158.803260:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134370944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697158.803267:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697158.803270:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697158.803273:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134370944:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697158.803277:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371072:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697158.803292:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697158.803299:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697158/real 1787697158] req@ffff9d85b9d0a300 x1874531134371072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697158.803306:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134371072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697158.803310:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697158.803312:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697158.803325:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697158.803349:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697158.803368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697158.803376:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697158/real 1787697158] req@ffff9d85b9d09c00 x1874531134371200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697158.803383:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134371200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697158.803391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697158.803393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697158.803394:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697158.803397:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697158.803407:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697158.803411:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697158/real 1787697158] req@ffff9d85b9d09f80 x1874531134371328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697158.803417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134371328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697158.803422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697158.803423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697158.803426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697158.803431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134371456:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697158.803460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697158.803472:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697158/real 1787697158] req@ffff9d85b9d08700 x1874531134371456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697158.803484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134371456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697158.803491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697158.803494:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697158.803498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134371456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697158.803503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134371584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697158.803522:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697158.803532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697158/real 1787697158] req@ffff9d85b9d0b800 x1874531134371584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697158.803543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134371584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697158.803548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697158.803550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697158.803552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134371584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697158.803555:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134371712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697158.803567:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697158.803573:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697158/real 1787697158] req@ffff9d85b9d09500 x1874531134371712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697158.803580:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134371712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697158.803584:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697158.803585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697158.803587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134371712:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697159.379020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697160.402996:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697161.426960:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697162.450983:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697163.475026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697163.922938:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697163.922945:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697163.922948:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697163.922956:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5065 00000100:00080000:0.0:1787697163.922965:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697163.922971:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697163.923002:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134371840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697163.923025:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697163.923028:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697163.923030:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697163.923033:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5065 00000100:00080000:0.0:1787697163.923037:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697163.923040:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697163.923049:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697163.923063:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134371968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697163.923072:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697163.923076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697163.923078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697163.923078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134371840:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697163.923082:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5065 00000100:00080000:0.0:1787697163.923089:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697163.923092:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697163.923095:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697163.923104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134372096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697163.923113:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697163.923116:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697163.923118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697163.923121:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5065 00000100:00080000:0.0:1787697163.923126:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697163.923130:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697163.923135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697163.923150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134372224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697163.923150:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697163.923159:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697163.923161:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697163.923163:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697163.923165:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5065 00000100:00080000:0.0:1787697163.923169:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697163.923172:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697163.923175:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697163.923182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134372352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697163.923188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697163.923190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697163.923193:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697163.923194:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697163/real 1787697163] req@ffff9d85b9d0b480 x1874531134371840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697163.923197:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5065 00000100:00080000:0.0:1787697163.923204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697163.923209:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697163.923213:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134371840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697163.923215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697163.923227:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697163.923230:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697163.923234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134372480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697163.923235:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134371840:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697163.923250:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697163.923253:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697163.923254:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697163.923258:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371968:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697163.923259:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5065 00000100:00080000:0.0:1787697163.923265:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697163.923269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697163.923274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697163.923283:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697163.923285:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134372608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697163.923295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697163/real 1787697163] req@ffff9d85b9d08e00 x1874531134371968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697163.923304:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134371968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697163.923309:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697163.923311:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697163.923313:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134371968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697163.923318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697163.923330:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697163.923339:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697163/real 1787697163] req@ffff9d85b9d08a80 x1874531134372096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697163.923347:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134372096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697163.923352:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697163.923354:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697163.923374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697163.923380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697163.923402:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697163.923413:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697163/real 1787697163] req@ffff9d85b9d08000 x1874531134372224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697163.923424:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134372224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697163.923431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697163.923434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697163.923437:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697163.923440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134372352:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697163.923454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697163.923460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697163/real 1787697163] req@ffff9d85b9d0bb80 x1874531134372352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697163.923467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134372352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697163.923482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697163.923484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697163.923487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134372352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697163.923490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134372480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697163.923500:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697163.923505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697163/real 1787697163] req@ffff9d85b9d0ad80 x1874531134372480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697163.923512:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134372480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697163.923516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697163.923517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697163.923519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134372480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697163.923530:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134372608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697163.923539:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697163.923544:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697163/real 1787697163] req@ffff9d85b9d0a680 x1874531134372608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697163.923550:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134372608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697163.923555:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697163.923556:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697163.923558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134372608:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697164.499021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697165.523005:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697166.547026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697167.570979:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697168.595022:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697169.042981:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697169.042992:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697169.042997:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697169.043007:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5070 00000100:00080000:2.0:1787697169.043018:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697169.043025:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697169.043064:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134372736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697169.043095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697169.043098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697169.043101:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697169.043105:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5070 00000100:00080000:2.0:1787697169.043108:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697169.043112:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697169.043134:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697169.043143:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134372736:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697169.043151:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134372864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697169.043164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697169.043174:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697169.043176:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697169.043180:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5070 00000100:00080000:2.0:1787697169.043193:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697169.043198:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697169.043203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697169.043214:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697169.043216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134372992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697169.043228:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697169.043232:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697169.043234:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697169.043237:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5070 00000100:00080000:2.0:1787697169.043241:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697169.043245:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697169.043250:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697169.043259:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697169/real 1787697169] req@ffff9d85babd4e00 x1874531134372736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697169.043263:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134373120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697169.043293:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697169.043297:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134372736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697169.043298:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697169.043302:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697169.043310:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5070 00000100:00080000:2.0:1787697169.043316:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697169.043317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697169.043322:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697169.043325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697169.043331:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697169.043332:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134372736:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697169.043348:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134373248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697169.043359:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697169.043361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372864:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697169.043365:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697169.043369:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697169.043373:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5070 00000100:00080000:2.0:1787697169.043377:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697169.043382:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697169.043387:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697169.043390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697169.043402:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134373376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697169.043407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697169/real 1787697169] req@ffff9d85babd5f80 x1874531134372864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697169.043418:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697169.043425:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697169.043426:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697169.043431:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5070 00000100:00100000:3.0:1787697169.043431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134372864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697169.043437:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697169.043442:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697169.043448:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697169.043451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697169.043457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787697169.043460:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134373504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697169.043465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697169.043476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697169.043498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697169.043508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697169/real 1787697169] req@ffff9d85babd4700 x1874531134372992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697169.043520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134372992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697169.043528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697169.043531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697169.043535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134372992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697169.043541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134373120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697169.043559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697169.043568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697169/real 1787697169] req@ffff9d85babd7100 x1874531134373120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697169.043580:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134373120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697169.043588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697169.043591:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697169.043594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134373120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697169.043609:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134373248:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697169.043626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697169.043636:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697169/real 1787697169] req@ffff9d85babd7b80 x1874531134373248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697169.043658:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134373248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697169.043666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697169.043670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697169.043673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134373248:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697169.043679:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134373376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697169.043696:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697169.043707:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697169/real 1787697169] req@ffff9d85babd4a80 x1874531134373376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697169.043718:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134373376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697169.043726:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697169.043728:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697169.043731:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134373376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697169.043737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134373504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697169.043754:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697169.043764:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697169/real 1787697169] req@ffff9d85babd5180 x1874531134373504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697169.043783:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134373504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697169.043799:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697169.043802:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697169.043805:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134373504:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697169.619009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697170.643022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697171.667047:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697172.690999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697173.714993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697174.162993:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697174.163004:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697174.163009:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697174.163021:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5075 00000100:00080000:0.0:1787697174.163034:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697174.163041:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697174.163093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134373632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697174.163125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697174.163128:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697174.163130:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697174.163134:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5075 00000100:00080000:0.0:1787697174.163137:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697174.163141:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697174.163152:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697174.163164:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134373760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697174.163175:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697174.163195:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697174.163197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697174.163201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5075 00000100:00080000:0.0:1787697174.163204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697174.163208:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697174.163214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697174.163217:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134373632:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697174.163226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134373888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697174.163237:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697174.163241:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697174.163242:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697174.163246:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5075 00000100:00080000:0.0:1787697174.163249:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697174.163254:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697174.163258:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697174.163269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134374016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697174.163278:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697174.163282:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697174.163284:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697174.163288:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5075 00000100:00080000:0.0:1787697174.163291:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697174.163296:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697174.163297:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697174.163304:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697174.163314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134374144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697174.163324:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697174.163328:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697174.163329:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697174.163333:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5075 00000100:00080000:0.0:1787697174.163336:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697174.163340:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697174.163346:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697174.163352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697174/real 1787697174] req@ffff9d85b9d0a680 x1874531134373632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697174.163358:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134374272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697174.163376:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697174.163380:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697174.163384:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697174.163384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134373632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697174.163392:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5075 00000100:00080000:0.0:1787697174.163398:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697174.163404:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697174.163405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787697174.163408:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697174.163413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697174.163419:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134373632:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697174.163422:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134374400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697174.163431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134373760:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697174.163455:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697174.163470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697174/real 1787697174] req@ffff9d85b9d0ad80 x1874531134373760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697174.163482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134373760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697174.163497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697174.163500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697174.163504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134373760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697174.163527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134373888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697174.163543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697174.163552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697174/real 1787697174] req@ffff9d85b9d0bb80 x1874531134373888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697174.163562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134373888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697174.163570:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697174.163573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697174.163576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134373888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697174.163582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697174.163595:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697174.163604:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697174/real 1787697174] req@ffff9d85b9d08000 x1874531134374016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697174.163614:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134374016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697174.163622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697174.163625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697174.163629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697174.163634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134374144:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697174.163665:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697174.163674:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697174/real 1787697174] req@ffff9d85b9d08a80 x1874531134374144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697174.163685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134374144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697174.163692:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697174.163695:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697174.163698:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134374144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697174.163703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134374272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697174.163718:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697174.163726:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697174/real 1787697174] req@ffff9d85b9d08e00 x1874531134374272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697174.163737:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134374272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697174.163744:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697174.163747:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697174.163750:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134374272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697174.163756:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134374400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697174.163768:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697174.163777:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697174/real 1787697174] req@ffff9d85b9d0b480 x1874531134374400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697174.163788:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134374400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697174.163795:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697174.163797:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697174.163800:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134374400:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697174.739031:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697175.763018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697176.786980:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697177.810998:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697178.834973:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697179.282929:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697179.282939:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697179.282942:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697179.282951:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5081 00000100:00080000:0.0:1787697179.282961:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697179.282967:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697179.283001:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134374528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697179.283025:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697179.283035:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697179.283037:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697179.283042:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5081 00000100:00080000:0.0:1787697179.283045:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697179.283049:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697179.283058:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697179.283069:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134374528:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697179.283071:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134374656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697179.283085:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697179.283088:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697179.283089:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697179.283103:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5081 00000100:00080000:0.0:1787697179.283106:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697179.283111:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697179.283116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697179.283127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134374784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697179.283135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697179.283149:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697179.283152:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697179.283155:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5081 00000100:00080000:0.0:1787697179.283162:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697179.283166:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697179.283171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697179.283183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134374912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697179.283190:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374656:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697179.283196:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697179.283200:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697179.283202:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697179.283207:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5081 00000100:00080000:0.0:1787697179.283212:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697179.283216:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697179.283221:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697179.283224:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697179.283234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134375040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697179.283244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697179.283247:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697179.283249:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697179.283253:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5081 00000100:00100000:3.0:1787697179.283257:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374912:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787697179.283258:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697179.283266:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697179.283271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697179.283283:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134375168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697179.283285:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375040:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697179.283298:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697179.283301:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697179.283303:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697179.283312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375168:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787697179.283320:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5081 00000100:00080000:0.0:1787697179.283325:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697179.283329:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697179.283334:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697179.283344:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134375296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697179.283375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375296:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697179.859040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697180.882974:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697181.907037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697182.355141:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697182.355181:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697182.355197:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697182.355236:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697182.355245:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697182.355257:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697182.355267:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787697182.355283:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697179/real 1787697182] req@ffff9d85b9d09f80 x1874531134374912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697182.355286:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697182.355298:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787697182.355316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134374912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697182.355331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697182.355336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697182.355343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697182.355350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697179/real 1787697182] req@ffff9d85b9d09c00 x1874531134375040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697182.355363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134375040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697182.355370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697182.355373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697182.355376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375040:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697182.355382:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697179/real 1787697182] req@ffff9d85b9d0b800 x1874531134374656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697182.355393:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134374656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697182.355400:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697182.355403:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697182.355406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697182.355412:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697179/real 1787697182] req@ffff9d85b9d08700 x1874531134374784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697182.355421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134374784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697182.355427:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697182.355430:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697182.355432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134374784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697182.355438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697179/real 1787697182] req@ffff9d85b9d09500 x1874531134374528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697182.355448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134374528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697182.355455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697182.355457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697182.355460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134374528:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697182.355487:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697179/real 1787697182] req@ffff9d85b9d09180 x1874531134375296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697182.355499:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134375296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697182.355506:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697182.355509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697182.355512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697182.355518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697179/real 1787697182] req@ffff9d85b9d0a300 x1874531134375168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697182.355528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134375168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697182.355549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697182.355553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697182.355556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375168:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697182.931074:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697183.955051:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697184.403087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697184.403101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697184.403106:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697184.403131:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5086 00000100:00080000:2.0:1787697184.403145:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697184.403153:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697184.403195:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134375424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697184.403229:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697184.403233:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697184.403236:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697184.403240:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5086 00000100:00080000:2.0:1787697184.403244:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697184.403248:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697184.403259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697184.403289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134375552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697184.403301:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697184.403312:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697184.403314:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697184.403319:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134375424:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697184.403320:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5086 00000100:00080000:2.0:1787697184.403329:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697184.403337:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697184.403343:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697184.403357:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134375680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697184.403369:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697184.403373:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697184.403375:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697184.403393:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5086 00000800:00000100:3.0:1787697184.404124:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697184.404186:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697184/real 1787697184] req@ffff9d85babd6a00 x1874531134375424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697184.404205:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134375424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697184.404215:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697184.404219:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697184.404223:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134375424:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697184.404260:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134375552:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697184.404638:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697184.404655:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697184.404668:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697184.404701:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134375808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697184.404717:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697184.404729:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697184.404731:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697184.404737:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5086 00000100:00080000:2.0:1787697184.404742:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697184.404746:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697184.404752:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697184.404765:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134375936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697184.404774:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697184.404777:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697184.404779:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697184.404783:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5086 00000100:00080000:2.0:1787697184.404786:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697184.404790:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697184.404795:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697184.404805:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134376064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697184.404813:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697184.404816:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697184.404817:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697184.404822:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5086 00000100:00080000:2.0:1787697184.404824:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697184.404828:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697184.404832:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697184.404842:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134376192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697184.406082:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697184.406110:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697184/real 1787697184] req@ffff9d85babd6680 x1874531134375552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697184.406127:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134375552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697184.406137:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697184.406142:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697184.406146:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134375552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697184.406155:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134375680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697184.406181:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697184.406192:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697184/real 1787697184] req@ffff9d85babd4000 x1874531134375680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697184.406204:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134375680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697184.406212:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697184.406216:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697184.406219:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134375680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697184.406241:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134375808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697184.406262:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697184.406273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697184/real 1787697184] req@ffff9d85babd4380 x1874531134375808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697184.406283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134375808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697184.406290:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697184.406293:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697184.406296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134375808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697184.406302:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375936:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697184.406321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697184.406332:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697184/real 1787697184] req@ffff9d85babd6300 x1874531134375936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697184.406343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134375936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697184.406351:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697184.406354:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697184.406357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134375936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697184.406363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697184.406382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697184.406393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697184/real 1787697184] req@ffff9d85babd7480 x1874531134376064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697184.406404:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134376064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697184.406411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697184.406415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697184.406418:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697184.406439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697184.406459:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697184.406471:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697184/real 1787697184] req@ffff9d85babd5c00 x1874531134376192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697184.406487:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134376192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697184.406494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697184.406497:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697184.406500:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376192:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697184.979156:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697186.003034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697187.027105:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697188.051052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697189.075004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697189.522933:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697189.522940:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697189.522943:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697189.522951:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5091 00000100:00080000:2.0:1787697189.522958:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697189.522964:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697189.522992:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134376320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697189.523012:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697189.523014:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697189.523016:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697189.523018:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5091 00000100:00080000:2.0:1787697189.523020:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697189.523022:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697189.523029:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697189.523037:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134376448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697189.523044:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697189.523046:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697189.523047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697189.523049:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5091 00000100:00080000:2.0:1787697189.523051:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697189.523054:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697189.523058:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697189.523058:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134376320:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697189.523066:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134376576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697189.523072:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697189.523074:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697189.523075:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697189.523077:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5091 00000100:00080000:2.0:1787697189.523079:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697189.523081:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697189.523083:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697189.523091:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134376704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697189.523096:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697189.523098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697189.523099:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697189.523101:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5091 00000100:00080000:2.0:1787697189.523111:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697189.523113:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697189.523116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697189.523123:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697189.523124:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134376832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697189.523131:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697189.523133:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697189.523135:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697189.523137:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5091 00000100:00080000:2.0:1787697189.523139:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697189.523141:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697189.523144:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697189.523149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134376960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697189.523154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697189.523156:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697189.523157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697189.523160:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5091 00000100:00080000:2.0:1787697189.523162:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697189.523164:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697189.523167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697189.523171:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697189/real 1787697189] req@ffff9d85babd6d80 x1874531134376320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697189.523176:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134377088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697189.523212:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134376320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697189.523224:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697189.523228:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697189.523232:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134376320:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697189.523239:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134376448:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697189.523261:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697189.523270:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697189/real 1787697189] req@ffff9d85babd5500 x1874531134376448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697189.523282:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134376448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697189.523287:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697189.523289:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697189.523291:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134376448:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697189.523307:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134376576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697189.523318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697189.523323:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697189/real 1787697189] req@ffff9d85babd7800 x1874531134376576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697189.523332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134376576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697189.523339:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697189.523342:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697189.523345:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134376576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697189.523351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134376704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697189.523376:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697189.523387:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697189/real 1787697189] req@ffff9d85babd5c00 x1874531134376704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697189.523398:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134376704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697189.523412:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697189.523415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697189.523417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134376704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697189.523421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376832:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697189.523435:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697189.523445:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697189/real 1787697189] req@ffff9d85babd7480 x1874531134376832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697189.523455:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134376832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697189.523462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697189.523465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697189.523468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697189.523474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697189.523491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697189.523502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697189/real 1787697189] req@ffff9d85babd6300 x1874531134376960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697189.523512:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134376960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697189.523520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697189.523523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697189.523525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134376960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697189.523529:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697189.523544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697189.523554:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697189/real 1787697189] req@ffff9d85babd4380 x1874531134377088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697189.523565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134377088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697189.523573:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697189.523576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697189.523578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377088:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697190.099016:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697191.122943:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697192.147053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697193.170958:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697194.195017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697194.642974:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697194.642984:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697194.642989:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697194.642997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5096 00000100:00080000:0.0:1787697194.643005:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697194.643011:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697194.643041:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134377216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697194.643059:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697194.643062:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697194.643063:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697194.643066:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5096 00000100:00080000:0.0:1787697194.643068:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697194.643070:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697194.643076:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697194.643084:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134377344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697194.643090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697194.643092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697194.643093:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697194.643095:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5096 00000100:00080000:0.0:1787697194.643097:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697194.643099:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697194.643103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697194.643108:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134377216:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697194.643109:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134377472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697194.643122:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697194.643124:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697194.643125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697194.643127:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5096 00000100:00080000:0.0:1787697194.643129:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697194.643131:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697194.643134:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697194.643140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134377600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697194.643145:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697194.643147:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697194.643149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697194.643151:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5096 00000100:00080000:0.0:1787697194.643154:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697194.643156:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697194.643158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697194.643163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134377728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697194.643168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697194.643170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697194.643171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697194.643174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5096 00000100:00080000:0.0:1787697194.643176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697194.643180:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697194.643181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697194.643185:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697194.643196:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134377856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697194.643205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697194.643207:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697194.643208:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697194.643210:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5096 00000100:00080000:0.0:1787697194.643212:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697194.643214:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697194.643216:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697194.643224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134377984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697194.643229:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697194/real 1787697194] req@ffff9d85b9d0a300 x1874531134377216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697194.643250:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134377216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697194.643260:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697194.643264:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697194.643267:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134377216:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697194.643279:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134377344:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697194.643301:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697194.643313:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697194/real 1787697194] req@ffff9d85b9d09180 x1874531134377344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697194.643325:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134377344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697194.643332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697194.643336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697194.643339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134377344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697194.643361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134377472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697194.643382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697194.643392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697194/real 1787697194] req@ffff9d85b9d09500 x1874531134377472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697194.643403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134377472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697194.643422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697194.643429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697194.643432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134377472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697194.643438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134377600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697194.643458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697194.643469:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697194/real 1787697194] req@ffff9d85b9d08700 x1874531134377600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697194.643482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134377600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697194.643490:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697194.643493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697194.643496:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134377600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697194.643502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377728:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697194.643521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697194.643532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697194/real 1787697194] req@ffff9d85b9d0b800 x1874531134377728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697194.643544:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134377728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697194.643551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697194.643554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697194.643557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697194.643563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697194.643580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697194.643591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697194/real 1787697194] req@ffff9d85b9d09c00 x1874531134377856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697194.643601:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134377856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697194.643615:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697194.643618:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697194.643621:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697194.643627:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697194.643642:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697194.643651:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697194/real 1787697194] req@ffff9d85b9d09f80 x1874531134377984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697194.643661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134377984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697194.643668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697194.643671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697194.643674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134377984:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697195.219009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697196.243000:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697197.267094:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697198.291088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697199.314990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697199.762902:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697199.762909:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697199.762912:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697199.762920:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5101 00000100:00080000:0.0:1787697199.762928:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697199.762933:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697199.762961:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134378112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697199.762979:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697199.762981:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697199.762983:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697199.762986:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5101 00000100:00080000:0.0:1787697199.762988:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697199.762990:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697199.762998:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697199.763006:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134378112:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697199.763007:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134378240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697199.763019:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697199.763021:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697199.763022:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697199.763024:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5101 00000100:00080000:0.0:1787697199.763026:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697199.763028:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697199.763030:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697199.763036:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134378368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697199.763042:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697199.763044:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697199.763045:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697199.763047:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5101 00000100:00080000:0.0:1787697199.763050:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697199.763052:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697199.763055:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697199.763061:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134378496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697199.763066:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697199.763068:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697199.763071:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697199.763073:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697199.763076:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5101 00000100:00080000:0.0:1787697199.763079:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697199.763082:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697199.763085:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697199.763091:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134378624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697199.763097:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697199.763099:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697199.763100:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697199.763103:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5101 00000100:00080000:0.0:1787697199.763105:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697199.763107:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697199.763110:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697199.763112:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697199/real 1787697199] req@ffff9d85b9d0b480 x1874531134378112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697199.763120:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134378752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697199.763128:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697199.763131:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787697199.763132:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134378112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697199.763133:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697199.763140:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5101 00000100:00080000:0.0:1787697199.763144:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697199.763154:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697199.763157:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697199.763160:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697199.763165:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697199.763169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134378880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697199.763171:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134378112:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697199.763197:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134378240:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697199.763213:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697199.763221:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697199/real 1787697199] req@ffff9d85b9d08e00 x1874531134378240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697199.763228:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134378240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697199.763233:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697199.763235:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697199.763237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134378240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697199.763240:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134378368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697199.763250:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697199.763255:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697199/real 1787697199] req@ffff9d85b9d08a80 x1874531134378368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697199.763261:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134378368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697199.763266:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697199.763267:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697199.763269:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134378368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697199.763272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134378496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697199.763281:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697199.763286:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697199/real 1787697199] req@ffff9d85b9d08000 x1874531134378496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697199.763292:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134378496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697199.763296:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697199.763298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697199.763299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134378496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697199.763302:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134378624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697199.763310:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697199.763314:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697199/real 1787697199] req@ffff9d85b9d0bb80 x1874531134378624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697199.763320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134378624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697199.763324:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697199.763326:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697199.763328:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134378624:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697199.763330:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134378752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697199.763338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697199.763342:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697199/real 1787697199] req@ffff9d85b9d0ad80 x1874531134378752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697199.763349:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134378752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697199.763353:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697199.763355:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697199.763356:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134378752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697199.763359:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134378880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697199.763367:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697199.763371:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697199/real 1787697199] req@ffff9d85b9d0a680 x1874531134378880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697199.763376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134378880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697199.763381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697199.763382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697199.763384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134378880:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697200.338963:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697201.363052:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697202.386981:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697203.411028:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697204.434993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697204.882966:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697204.882975:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697204.882980:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697204.882990:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5106 00000100:00080000:0.0:1787697204.883001:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697204.883008:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697204.883048:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134379008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697204.883071:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697204.883074:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697204.883075:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697204.883078:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5106 00000100:00080000:0.0:1787697204.883080:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697204.883082:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697204.883091:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697204.883100:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134379136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697204.883107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697204.883109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697204.883110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697204.883115:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5106 00000100:00080000:0.0:1787697204.883116:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697204.883116:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134379008:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697204.883123:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697204.883128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697204.883135:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134379264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697204.883141:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697204.883143:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697204.883144:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697204.883146:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5106 00000100:00080000:0.0:1787697204.883148:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697204.883150:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697204.883153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697204.883160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134379392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697204.883166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697204.883168:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697204.883169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697204.883172:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5106 00000100:00080000:0.0:1787697204.883174:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697204.883176:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697204.883178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697204.883184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134379520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697204.883188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697204.883192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787697204.883194:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697204.883195:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697204.883201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5106 00000100:00080000:0.0:1787697204.883203:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697204.883206:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697204.883209:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697204.883216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134379648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697204.883221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697204.883224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697204.883225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697204.883228:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5106 00000100:00080000:0.0:1787697204.883229:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697204.883232:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697204.883235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697204.883241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697204/real 1787697204] req@ffff9d85b9d0a680 x1874531134379008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697204.883244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134379776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697204.883278:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134379008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697204.883287:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697204.883290:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697204.883294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134379008:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697204.883315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134379136:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697204.883330:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697204.883337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697204/real 1787697204] req@ffff9d85b9d0ad80 x1874531134379136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697204.883345:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134379136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697204.883350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697204.883351:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697204.883353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134379136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697204.883356:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134379264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697204.883366:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697204.883371:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697204/real 1787697204] req@ffff9d85b9d0bb80 x1874531134379264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697204.883378:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134379264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697204.883386:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697204.883388:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697204.883390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134379264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697204.883393:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134379392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697204.883405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697204.883414:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697204/real 1787697204] req@ffff9d85b9d08000 x1874531134379392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697204.883424:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134379392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697204.883431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697204.883434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697204.883438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134379392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697204.883444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134379520:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697204.883462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697204.883473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697204/real 1787697204] req@ffff9d85b9d08a80 x1874531134379520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697204.883484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134379520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697204.883492:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697204.883495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697204.883498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134379520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697204.883502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134379648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697204.883521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697204.883531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697204/real 1787697204] req@ffff9d85b9d08e00 x1874531134379648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697204.883543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134379648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697204.883550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697204.883552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697204.883555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134379648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697204.883558:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134379776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697204.883572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697204.883579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697204/real 1787697204] req@ffff9d85b9d0b480 x1874531134379776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697204.883590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134379776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697204.883598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697204.883601:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697204.883604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134379776:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697205.459070:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697206.483038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697207.507076:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697208.531053:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697209.555089:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697210.002978:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697210.002989:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697210.002994:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697210.003004:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5111 00000100:00080000:0.0:1787697210.003015:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697210.003024:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697210.003066:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134379904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697210.003100:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697210.003105:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697210.003107:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697210.003112:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5111 00000100:00080000:0.0:1787697210.003115:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697210.003119:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697210.003129:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697210.003143:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134380032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697210.003150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134379904:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697210.003162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697210.003166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697210.003168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697210.003173:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5111 00000100:00080000:0.0:1787697210.003177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697210.003182:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697210.003188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697210.003201:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134380160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697210.003212:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697210.003214:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697210.003219:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697210.003222:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697210.003228:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5111 00000100:00080000:0.0:1787697210.003235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697210.003241:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697210.003247:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697210.003257:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697210/real 1787697210] req@ffff9d85b9d09f80 x1874531134379904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697210.003264:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134380288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697210.003276:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134379904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697210.003278:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697210.003288:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697210.003294:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697210.003296:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697210.003302:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5111 00000100:00080000:3.0:1787697210.003302:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697210.003308:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134379904:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697210.003310:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697210.003318:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697210.003323:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697210.003329:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697210.003336:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134380416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697210.003346:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697210.003351:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787697210.003351:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697210.003353:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697210.003358:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5111 00000100:00080000:0.0:1787697210.003363:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697210.003367:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697210/real 1787697210] req@ffff9d85b9d09c00 x1874531134380032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697210.003369:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697210.003376:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697210.003385:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134380032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697210.003390:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134380544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697210.003402:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697210.003409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697210.003417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380032:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697210.003422:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697210.003426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380160:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787697210.003427:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697210.003430:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697210.003436:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5111 00000100:00080000:0.0:1787697210.003439:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697210.003442:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697210.003446:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697210.003451:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697210.003459:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134380672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697210.003463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697210/real 1787697210] req@ffff9d85b9d0b800 x1874531134380160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697210.003480:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134380160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697210.003486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697210.003488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697210.003491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697210.003496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697210.003507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697210.003513:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697210/real 1787697210] req@ffff9d85b9d08700 x1874531134380288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697210.003520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134380288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697210.003526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697210.003527:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697210.003529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697210.003539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134380416:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697210.003549:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697210.003555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697210/real 1787697210] req@ffff9d85b9d09500 x1874531134380416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697210.003587:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134380416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697210.003597:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697210.003599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697210.003602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134380416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697210.003606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134380544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697210.003619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697210.003626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697210/real 1787697210] req@ffff9d85b9d09180 x1874531134380544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697210.003633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134380544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697210.003639:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697210.003640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697210.003642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134380544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697210.003646:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134380672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697210.003655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697210.003661:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697210/real 1787697210] req@ffff9d85b9d0a300 x1874531134380672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697210.003668:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134380672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697210.003673:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697210.003674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697210.003677:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134380672:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697210.579056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697211.603088:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697212.626973:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697213.651017:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697214.675021:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697215.122942:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697215.122948:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697215.122952:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697215.122960:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5116 00000100:00080000:2.0:1787697215.122969:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697215.122975:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697215.123018:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134380800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697215.123044:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697215.123047:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697215.123049:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697215.123052:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5116 00000100:00080000:2.0:1787697215.123055:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697215.123059:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697215.123068:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697215.123080:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134380800:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697215.123082:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134380928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697215.123098:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697215.123101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697215.123102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697215.123105:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5116 00000100:00080000:2.0:1787697215.123107:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697215.123110:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697215.123114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697215.123122:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134381056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697215.123129:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697215.123131:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697215.123132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697215.123134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5116 00000100:00080000:2.0:1787697215.123137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697215.123141:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697215.123143:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697215.123151:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134381184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697215.123151:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697215.123160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697215.123163:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697215.123165:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697215.123167:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5116 00000100:00080000:2.0:1787697215.123171:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697215.123174:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697215.123177:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697215.123184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134381312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697215.123190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697215.123192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697215.123193:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697215.123197:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5116 00000100:00080000:2.0:1787697215.123199:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697215.123202:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697215/real 1787697215] req@ffff9d85babd4000 x1874531134380800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697215.123205:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697215.123212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697215.123225:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134381440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697215.123231:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134380800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697215.123237:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697215.123245:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697215.123247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697215.123254:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5116 00000100:00080000:3.0:1787697215.123255:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697215.123257:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697215.123262:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697215.123263:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697215.123270:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134380800:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787697215.123276:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697215.123286:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134381568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697215.123295:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380928:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697215.123323:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697215.123336:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697215/real 1787697215] req@ffff9d85babd6680 x1874531134380928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697215.123345:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134380928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697215.123351:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697215.123353:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697215.123355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134380928:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697215.123361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697215.123372:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697215.123379:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697215/real 1787697215] req@ffff9d85babd6a00 x1874531134381056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697215.123386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134381056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697215.123391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697215.123393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697215.123395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697215.123398:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697215.123407:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697215.123413:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697215/real 1787697215] req@ffff9d85babd5180 x1874531134381184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697215.123420:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134381184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697215.123425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697215.123427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697215.123428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697215.123432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134381312:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697215.123445:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697215.123456:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697215/real 1787697215] req@ffff9d85babd4a80 x1874531134381312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697215.123469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134381312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697215.123494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697215.123498:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697215.123501:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134381312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697215.123507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134381440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697215.123523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697215.123534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697215/real 1787697215] req@ffff9d85babd7b80 x1874531134381440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697215.123546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134381440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697215.123554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697215.123558:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697215.123561:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134381440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697215.123581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134381568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697215.123597:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697215.123605:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697215/real 1787697215] req@ffff9d85babd7100 x1874531134381568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697215.123612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134381568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697215.123622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697215.123624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697215.123626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134381568:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697215.698999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697216.723015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697217.747015:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697218.771041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697219.795014:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697220.242990:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697220.243001:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697220.243006:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697220.243019:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5122 00000100:00080000:1.0:1787697220.243031:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697220.243040:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697220.243078:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134381696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697220.243107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697220.243114:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697220.243116:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697220.243121:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5122 00000100:00080000:1.0:1787697220.243125:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697220.243131:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697220.243142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697220.243148:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134381696:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697220.243163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134381824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697220.243178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697220.243182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697220.243184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697220.243190:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5122 00000100:00080000:1.0:1787697220.243195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697220.243200:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697220.243207:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697220.243212:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697220.243224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134381952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697220.243237:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697220.243241:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697220.243243:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697220.243247:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5122 00000100:00080000:1.0:1787697220.243251:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697220.243256:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697220.243260:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697220/real 1787697220] req@ffff9d85b9d2dc00 x1874531134381696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697220.243265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697220.243280:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134381696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697220.243288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134382080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697220.243291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697220.243298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697220.243306:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134381696:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697220.243315:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697220.243319:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697220.243322:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697220.243327:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5122 00000100:00080000:1.0:1787697220.243331:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697220.243333:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381824:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697220.243342:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697220.243349:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697220.243357:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697220.243366:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134382208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697220.243370:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697220/real 1787697220] req@ffff9d85b9d2f800 x1874531134381824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697220.243408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134381824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697220.243418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697220.243424:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697220.243430:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381824:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697220.243434:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697220.243438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381952:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787697220.243442:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697220.243451:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697220.243459:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5122 00000800:00000100:3.0:1787697220.243462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697220.243471:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697220.243480:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697220.243484:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697220/real 1787697220] req@ffff9d85b9d2c380 x1874531134381952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697220.243489:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697220.243498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134381952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697220.243505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697220.243511:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134382336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697220.243511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697220.243517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134381952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697220.243527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382080:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787697220.243539:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697220.243543:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697220.243547:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697220.243550:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697220.243558:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5122 00000100:00080000:1.0:1787697220.243564:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697220.243569:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697220/real 1787697220] req@ffff9d85b9d2fb80 x1874531134382080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697220.243571:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697220.243582:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697220.243587:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134382080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697220.243594:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697220.243597:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697220.243604:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134382464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697220.243604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697220.243626:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134382208:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697220.243644:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697220.243654:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697220/real 1787697220] req@ffff9d85b9d2e300 x1874531134382208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697220.243662:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134382208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697220.243667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697220.243669:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697220.243670:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134382208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697220.243675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134382336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697220.243693:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697220.243707:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697220/real 1787697220] req@ffff9d85b9d2ce00 x1874531134382336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697220.243723:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134382336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697220.243732:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697220.243736:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697220.243738:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134382336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697220.243743:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134382464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697220.243757:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697220.243765:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697220/real 1787697220] req@ffff9d85b9d2c000 x1874531134382464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697220.243773:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134382464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697220.243778:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697220.243780:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697220.243782:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134382464:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697220.819056:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697221.843044:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697222.867060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697223.891079:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697224.915069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697225.363014:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697225.363025:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697225.363029:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697225.363039:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5127 00000100:00080000:2.0:1787697225.363052:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697225.363059:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697225.363101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134382592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697225.363126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697225.363129:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697225.363130:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697225.363133:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5127 00000100:00080000:2.0:1787697225.363135:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697225.363137:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697225.363146:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697225.363155:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134382720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697225.363162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697225.363164:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697225.363166:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697225.363169:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5127 00000100:00080000:2.0:1787697225.363171:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697225.363174:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697225.363178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697225.363184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134382848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697225.363191:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697225.363193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697225.363194:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697225.363196:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5127 00000100:00080000:2.0:1787697225.363198:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697225.363201:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697225.363203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697225.363211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134382976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697225.363219:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697225.363222:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697225.363223:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697225.363226:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134382592:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697225.363228:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5127 00000100:00080000:2.0:1787697225.363231:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697225.363238:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697225.363247:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697225.363256:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134383104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697225.363261:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697225.363263:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697225.363264:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697225.363266:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5127 00000100:00080000:2.0:1787697225.363268:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697225.363271:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697225.363273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697225.363280:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134383232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697225.363285:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697225.363287:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697225.363288:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697225.363291:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5127 00000100:00080000:2.0:1787697225.363322:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697225.363327:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697225.363331:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697225.363339:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697225.363343:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134383360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697225.363403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697225/real 1787697225] req@ffff9d85babd4700 x1874531134382592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697225.363426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134382592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697225.363438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697225.363443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697225.363447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134382592:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697225.363457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382720:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697225.363483:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697225.363496:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697225/real 1787697225] req@ffff9d85babd5f80 x1874531134382720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697225.363517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134382720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697225.363526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697225.363529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697225.363533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697225.363540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697225.363559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697225.363571:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697225/real 1787697225] req@ffff9d85babd4e00 x1874531134382848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697225.363584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134382848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697225.363592:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697225.363596:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697225.363599:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697225.363637:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697225.363659:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697225.363671:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697225/real 1787697225] req@ffff9d85babd7100 x1874531134382976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697225.363684:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134382976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697225.363699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697225.363703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697225.363706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134382976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697225.363713:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134383104:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697225.363743:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697225.363754:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697225/real 1787697225] req@ffff9d85babd7b80 x1874531134383104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697225.363767:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134383104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697225.363776:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697225.363779:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697225.363783:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134383104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697225.363790:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134383232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697225.363820:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697225.363832:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697225/real 1787697225] req@ffff9d85babd4a80 x1874531134383232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697225.363845:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134383232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697225.363897:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697225.363901:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697225.363906:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134383232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697225.363912:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134383360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697225.363935:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697225.363948:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697225/real 1787697225] req@ffff9d85babd5180 x1874531134383360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697225.363970:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134383360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697225.363979:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697225.363983:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697225.363986:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134383360:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697225.939121:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697226.963112:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697227.987102:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697229.011107:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697230.035042:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697230.482969:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697230.482979:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697230.482984:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697230.482994:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5132 00000100:00080000:1.0:1787697230.483007:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697230.483013:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697230.483061:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134383488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697230.483085:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697230.483088:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697230.483089:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697230.483092:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5132 00000100:00080000:1.0:1787697230.483094:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697230.483097:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697230.483107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697230.483116:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134383616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697230.483128:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697230.483130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697230.483131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697230.483134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5132 00000100:00080000:1.0:1787697230.483137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697230.483142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697230.483149:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697230.483150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134383488:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697230.483161:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134383744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697230.483168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697230.483171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697230.483172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697230.483175:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5132 00000100:00080000:1.0:1787697230.483176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697230.483178:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697230.483182:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697230.483191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134383872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697230.483197:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697230.483199:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697230.483200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697230.483203:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5132 00000100:00080000:1.0:1787697230.483230:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697230.483235:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697230.483239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697230.483251:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134384000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697230.483254:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697230.483268:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697230.483271:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697230.483273:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697230.483278:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5132 00000100:00080000:1.0:1787697230.483284:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697230.483288:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697230.483293:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697230.483305:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134384128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697230.483314:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697230.483318:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697230.483320:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697230.483325:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5132 00000100:00100000:3.0:1787697230.483331:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697230/real 1787697230] req@ffff9d85b9d2f480 x1874531134383488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697230.483332:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697230.483343:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697230.483350:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697230.483374:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134384256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697230.483379:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134383488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697230.483429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697230.483434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697230.483438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134383488:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697230.483475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134383616:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697230.483501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697230.483513:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697230/real 1787697230] req@ffff9d85b9d2ca80 x1874531134383616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697230.483524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134383616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697230.483540:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697230.483544:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697230.483547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134383616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697230.483553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134383744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697230.483572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697230.483590:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697230/real 1787697230] req@ffff9d85b9d2c000 x1874531134383744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697230.483601:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134383744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697230.483608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697230.483611:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697230.483615:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134383744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697230.483621:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134383872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697230.483638:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697230.483648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697230/real 1787697230] req@ffff9d85b9d2ce00 x1874531134383872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697230.483660:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134383872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697230.483668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697230.483671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697230.483674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134383872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697230.483679:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384000:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697230.483696:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697230.483705:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697230/real 1787697230] req@ffff9d85b9d2e300 x1874531134384000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697230.483715:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134384000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697230.483722:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697230.483725:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697230.483728:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697230.483734:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697230.483749:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697230.483759:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697230/real 1787697230] req@ffff9d85b9d2fb80 x1874531134384128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697230.483769:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134384128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697230.483782:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697230.483785:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697230.483788:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697230.483794:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697230.483810:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697230.483819:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697230/real 1787697230] req@ffff9d85b9d2c380 x1874531134384256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697230.483830:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134384256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697230.483837:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697230.483840:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697230.483843:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384256:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697231.059059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697232.083028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697233.107056:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697234.131085:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697235.155071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697235.602989:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697235.603000:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697235.603004:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697235.603014:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5137 00000100:00080000:0.0:1787697235.603025:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697235.603032:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697235.603066:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134384384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697235.603094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697235.603098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697235.603100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697235.603104:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5137 00000100:00080000:0.0:1787697235.603107:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697235.603110:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697235.603132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697235.603147:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134384512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697235.603157:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134384384:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697235.603160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697235.603165:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697235.603168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697235.603172:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5137 00000100:00080000:0.0:1787697235.603176:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697235.603180:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697235.603186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697235.603207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134384640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697235.603208:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697235.603220:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697235.603223:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697235.603225:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697235.603230:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5137 00000100:00080000:0.0:1787697235.603233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697235.603237:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697235.603241:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697235.603244:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697235/real 1787697235] req@ffff9d85b9d0a300 x1874531134384384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697235.603255:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134384768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697235.603259:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134384384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697235.603277:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697235.603283:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697235.603283:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697235.603288:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697235.603289:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697235.603294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134384384:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697235.603304:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5137 00000100:00080000:0.0:1787697235.603308:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697235.603312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134384512:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697235.603315:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697235.603322:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697235.603333:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697235.603337:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134384896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697235.603344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697235/real 1787697235] req@ffff9d85b9d09180 x1874531134384512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697235.603353:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697235.603357:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697235.603360:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697235.603369:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5137 00000100:00080000:0.0:1787697235.603374:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697235.603374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134384512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697235.603383:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697235.603383:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697235.603389:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:0.0:1787697235.603391:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697235.603395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134384512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697235.603404:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134384640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697235.603415:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134385024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697235.603426:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697235.603429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697235.603432:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697235.603436:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697235.603441:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5137 00000100:00080000:0.0:1787697235.603445:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697235.603445:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697235/real 1787697235] req@ffff9d85b9d09500 x1874531134384640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697235.603452:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697235.603459:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697235.603461:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134384640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697235.603469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697235.603472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697235.603476:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134385152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697235.603477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134384640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697235.603488:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134384768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697235.603505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697235.603512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697235/real 1787697235] req@ffff9d85b9d08700 x1874531134384768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697235.603520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134384768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697235.603525:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697235.603527:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697235.603528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134384768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697235.603538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384896:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697235.603547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697235.603552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697235/real 1787697235] req@ffff9d85b9d0b800 x1874531134384896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697235.603559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134384896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697235.603563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697235.603564:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697235.603566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134384896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697235.603569:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697235.603577:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697235.603582:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697235/real 1787697235] req@ffff9d85b9d09c00 x1874531134385024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697235.603589:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134385024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697235.603593:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697235.603595:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697235.603596:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697235.603599:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697235.603606:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697235.603611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697235/real 1787697235] req@ffff9d85b9d09f80 x1874531134385152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697235.603617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134385152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697235.603622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697235.603623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697235.603624:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385152:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697236.179079:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697237.203046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697238.227116:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697239.251129:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697240.275035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697240.722946:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697240.722955:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697240.722959:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697240.722967:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5142 00000100:00080000:0.0:1787697240.722979:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697240.722986:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697240.723022:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134385280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697240.723055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697240.723058:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697240.723059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697240.723062:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5142 00000100:00080000:0.0:1787697240.723065:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697240.723068:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697240.723076:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697240.723086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134385408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697240.723093:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697240.723096:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697240.723098:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697240.723103:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5142 00000100:00100000:3.0:1787697240.723106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134385280:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697240.723107:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697240.723116:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697240.723121:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697240.723134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134385536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697240.723143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697240.723145:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697240.723147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697240.723149:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5142 00000100:00080000:0.0:1787697240.723152:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697240.723154:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697240.723157:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697240.723165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134385664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697240.723170:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697240.723172:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697240.723175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697240.723177:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697240.723179:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5142 00000100:00080000:0.0:1787697240.723182:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697240.723185:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697240.723187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697240.723194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134385792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697240.723200:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697240.723202:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697240.723203:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697240.723205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5142 00000100:00080000:0.0:1787697240.723208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697240.723210:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697240.723213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697240.723220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134385920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697240.723226:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697240/real 1787697240] req@ffff9d85b9d0b480 x1874531134385280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697240.723227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697240.723233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697240.723235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697240.723242:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5142 00000100:00080000:0.0:1787697240.723247:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697240.723251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697240.723257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697240.723266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134386048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697240.723273:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134385280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697240.723283:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697240.723288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697240.723292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134385280:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697240.723299:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134385408:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697240.723321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697240.723332:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697240/real 1787697240] req@ffff9d85b9d08e00 x1874531134385408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697240.723343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134385408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697240.723350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697240.723352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697240.723355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134385408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697240.723377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134385536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697240.723397:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697240.723408:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697240/real 1787697240] req@ffff9d85b9d08a80 x1874531134385536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697240.723419:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134385536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697240.723430:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697240.723433:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697240.723436:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134385536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697240.723440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134385664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697240.723457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697240.723468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697240/real 1787697240] req@ffff9d85b9d08000 x1874531134385664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697240.723477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134385664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697240.723495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697240.723498:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697240.723501:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134385664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697240.723506:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385792:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697240.723524:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697240.723533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697240/real 1787697240] req@ffff9d85b9d0bb80 x1874531134385792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697240.723555:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134385792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697240.723562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697240.723565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697240.723569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697240.723574:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697240.723593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697240.723603:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697240/real 1787697240] req@ffff9d85b9d0ad80 x1874531134385920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697240.723614:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134385920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697240.723620:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697240.723623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697240.723627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134385920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697240.723632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697240.723650:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697240.723661:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697240/real 1787697240] req@ffff9d85b9d0a680 x1874531134386048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697240.723673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134386048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697240.723680:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697240.723683:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697240.723686:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386048:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697241.299045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697242.323019:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697243.347048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697244.371084:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697245.395038:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697245.842928:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697245.842937:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697245.842954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697245.842964:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5147 00000100:00080000:0.0:1787697245.842975:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697245.842982:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697245.843031:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134386176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697245.843057:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697245.843061:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697245.843063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697245.843067:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5147 00000100:00080000:0.0:1787697245.843070:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697245.843075:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697245.843084:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697245.843098:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134386176:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697245.843099:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134386304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697245.843114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697245.843118:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697245.843120:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697245.843124:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5147 00000100:00080000:0.0:1787697245.843127:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697245.843130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697245.843136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697245.843146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134386432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697245.843156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697245.843159:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697245.843161:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697245.843164:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5147 00000100:00080000:0.0:1787697245.843169:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697245.843172:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697245.843186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697245.843194:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134386304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697245.843201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134386560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697245.843212:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697245.843216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787697245.843219:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134386432:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787697245.843220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697245.843229:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5147 00000100:00080000:0.0:1787697245.843233:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697245.843238:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697245.843243:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134386560:192.168.201.137@tcp:8:kworker.0 00000008:00100000:0.0:1787697245.843246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697245.843261:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134386688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697245.843273:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697245.843278:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697245.843280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697245.843284:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5147 00000100:00100000:3.0:1787697245.843288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386688:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697245.843290:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697245.843297:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697245.843304:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697245.843316:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134386816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697245.843336:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697245.843340:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697245.843342:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697245.843345:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5147 00000100:00080000:0.0:1787697245.843348:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697245.843353:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697245.843357:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697245.843367:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697245.843371:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134386944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697245.843401:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386944:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697246.419053:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697247.443037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697248.467012:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697248.915115:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697248.915152:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697248.915166:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697248.915211:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697248.915220:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697248.915229:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697248.915237:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697248.915246:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697248.915256:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787697248.915286:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697245/real 1787697248] req@ffff9d85b9d08e00 x1874531134386816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697248.915316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134386816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697248.915341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697248.915346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697248.915351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697248.915360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697245/real 1787697248] req@ffff9d85b9d08a80 x1874531134386688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697248.915373:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134386688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697248.915381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697248.915385:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697248.915400:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697248.915407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697245/real 1787697248] req@ffff9d85b9d08000 x1874531134386560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697248.915418:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134386560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697248.915426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697248.915429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697248.915432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134386560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697248.915437:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697245/real 1787697248] req@ffff9d85b9d0ad80 x1874531134386304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697248.915447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134386304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697248.915453:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697248.915456:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697248.915459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134386304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697248.915465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697245/real 1787697248] req@ffff9d85b9d0bb80 x1874531134386432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697248.915475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134386432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697248.915482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697248.915485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697248.915488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134386432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697248.915494:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697245/real 1787697248] req@ffff9d85b9d0a680 x1874531134386176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697248.915504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134386176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697248.915511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697248.915514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697248.915518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134386176:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697248.915549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697245/real 1787697248] req@ffff9d85b9d0b480 x1874531134386944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697248.915561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134386944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697248.915569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697248.915573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697248.915576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134386944:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697249.490989:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697250.515045:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697250.963015:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697250.963029:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697250.963034:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697250.963047:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5152 00000100:00080000:0.0:1787697250.963061:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697250.963069:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697250.963113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134387072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697250.963156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697250.963160:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697250.963162:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697250.963167:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5152 00000100:00080000:0.0:1787697250.963170:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697250.963175:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697250.963188:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697250.963202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134387200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697250.963214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697250.963217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697250.963219:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697250.963223:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5152 00000100:00080000:0.0:1787697250.963227:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697250.963233:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697250.963234:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134387072:192.168.201.137@tcp:250:kworker.0 00000008:00100000:0.0:1787697250.963239:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697250.963250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134387328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697250.963261:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697250.963265:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697250.963267:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697250.963271:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5152 00000100:00080000:0.0:1787697250.963274:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697250.963279:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697250.963284:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697250.963296:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134387456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697250.963305:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697250.963309:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697250.963311:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697250.963315:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5152 00000100:00080000:0.0:1787697250.963327:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697250.963332:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697250.963338:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697250.963343:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697250.963354:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134387584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697250.963366:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697250.963371:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697250.963373:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697250.963378:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5152 00000100:00080000:0.0:1787697250.963381:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697250.963385:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697250.963390:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697250.963403:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134387712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697250.963413:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697250.963418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697250/real 1787697250] req@ffff9d85b9d09f80 x1874531134387072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697250.963419:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697250.963422:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697250.963430:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5152 00000100:00080000:0.0:1787697250.963435:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697250.963441:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697250.963446:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697250.963450:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134387072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697250.963462:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134387840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697250.963476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697250.963483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697250.963488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134387072:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697250.963496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134387200:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697250.963522:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697250.963535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697250/real 1787697250] req@ffff9d85b9d09c00 x1874531134387200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697250.963546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134387200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697250.963553:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697250.963556:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697250.963558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134387200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697250.963587:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134387328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697250.963610:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697250.963623:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697250/real 1787697250] req@ffff9d85b9d0b800 x1874531134387328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697250.963635:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134387328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697250.963648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697250.963651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697250.963654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134387328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697250.963660:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134387456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697250.963677:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697250.965301:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697250/real 1787697250] req@ffff9d85b9d08700 x1874531134387456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697250.965333:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134387456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697250.965347:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697250.965352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697250.965356:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134387456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697250.965368:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134387584:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697250.965446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697250.965461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697250/real 1787697250] req@ffff9d85b9d09500 x1874531134387584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697250.965473:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134387584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697250.965481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697250.965485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697250.965489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134387584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697250.965497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134387712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697250.965519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697250.965529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697250/real 1787697250] req@ffff9d85b9d09180 x1874531134387712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697250.965541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134387712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697250.965549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697250.965552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697250.965556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134387712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697250.965562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134387840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697250.965579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697250.965589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697250/real 1787697250] req@ffff9d85b9d0a300 x1874531134387840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697250.965601:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134387840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697250.965609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697250.965613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697250.965616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134387840:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697251.539003:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697252.562984:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697253.587004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697254.610991:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697255.635044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697256.082947:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697256.082956:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697256.082959:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697256.082967:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5157 00000100:00080000:1.0:1787697256.082976:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697256.082981:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697256.083010:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134387968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697256.083029:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697256.083032:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697256.083033:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697256.083036:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5157 00000100:00080000:1.0:1787697256.083038:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697256.083040:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697256.083046:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697256.083054:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134388096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697256.083061:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697256.083063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697256.083065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697256.083067:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5157 00000100:00080000:1.0:1787697256.083069:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697256.083073:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083074:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134387968:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787697256.083077:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697256.083084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134388224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697256.083090:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697256.083092:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697256.083093:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697256.083095:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5157 00000100:00080000:1.0:1787697256.083097:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697256.083099:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697256.083102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697256.083108:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134388352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697256.083113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697256.083115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697256.083117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697256.083119:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5157 00000100:00080000:1.0:1787697256.083121:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697256.083123:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697256.083126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697256.083129:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697256.083135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134388480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697256.083141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697256.083143:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697256.083144:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697256.083148:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5157 00000100:00080000:1.0:1787697256.083150:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697256.083152:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697256.083154:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697256.083160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134388608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697256.083164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697256.083166:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697256.083167:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697256.083170:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5157 00000100:00080000:1.0:1787697256.083173:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697256.083177:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083177:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697256/real 1787697256] req@ffff9d85b9d2f800 x1874531134387968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697256.083182:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697256.083197:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134388736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697256.083202:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134387968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697256.083222:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697256.083227:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697256.083231:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134387968:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697256.083238:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388096:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697256.083260:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083272:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697256/real 1787697256] req@ffff9d85b9d2dc00 x1874531134388096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697256.083280:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134388096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697256.083285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697256.083287:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697256.083289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697256.083306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697256.083319:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083325:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697256/real 1787697256] req@ffff9d85b9d2e680 x1874531134388224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697256.083332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134388224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697256.083337:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697256.083338:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697256.083340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697256.083343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697256.083351:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083356:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697256/real 1787697256] req@ffff9d85b9d2d500 x1874531134388352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697256.083362:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134388352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697256.083366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697256.083368:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697256.083369:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697256.083372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134388480:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697256.083380:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083385:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697256/real 1787697256] req@ffff9d85b9d2ed80 x1874531134388480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697256.083391:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134388480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697256.083399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697256.083400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697256.083402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134388480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697256.083405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134388608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697256.083412:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083417:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697256/real 1787697256] req@ffff9d85b9d2c700 x1874531134388608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697256.083422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134388608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697256.083426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697256.083429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697256.083430:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134388608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697256.083433:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134388736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697256.083441:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697256.083446:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697256/real 1787697256] req@ffff9d85b9d2c380 x1874531134388736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697256.083451:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134388736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697256.083455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697256.083457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697256.083459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134388736:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697256.658973:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697257.682998:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697258.707054:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697259.730990:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697260.755051:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697261.202985:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697261.202999:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697261.203005:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697261.203018:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5162 00000100:00080000:1.0:1787697261.203032:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697261.203040:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697261.203090:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134388864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697261.203128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697261.203132:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697261.203134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697261.203138:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5162 00000100:00080000:1.0:1787697261.203142:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697261.203145:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697261.203156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697261.203171:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134388992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697261.203181:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697261.203186:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134388864:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697261.203187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697261.203190:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697261.203198:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5162 00000100:00080000:1.0:1787697261.203202:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697261.203206:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697261.203212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697261.203227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134389120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697261.203237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697261.203241:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697261.203243:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697261.203247:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5162 00000100:00080000:1.0:1787697261.203260:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697261.203264:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697261.203269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697261.203276:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697261.203284:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134389248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697261.203295:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697261.203299:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697261.203301:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697261.203305:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5162 00000100:00080000:1.0:1787697261.203309:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697261.203313:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697261.203319:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697261.203329:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697261/real 1787697261] req@ffff9d85b9d2fb80 x1874531134388864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697261.203333:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134389376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697261.203356:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697261.203360:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697261.203362:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697261.203365:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134388864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697261.203369:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5162 00000100:00080000:1.0:1787697261.203376:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697261.203383:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697261.203385:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697261.203390:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697261.203392:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697261.203398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134388864:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697261.203416:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134389504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697261.203426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388992:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697261.203443:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697261.203447:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697261.203448:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697261.203452:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5162 00000800:00000100:3.0:1787697261.203457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697261.203459:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697261.203464:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697261.203469:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697261.203474:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697261/real 1787697261] req@ffff9d85b9d2e300 x1874531134388992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697261.203481:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134389632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697261.203502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134388992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697261.203512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697261.203516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697261.203519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134388992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697261.203527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134389120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697261.203546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697261.203556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697261/real 1787697261] req@ffff9d85b9d2ce00 x1874531134389120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697261.203568:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134389120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697261.203576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697261.203579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697261.203583:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134389120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697261.203588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134389248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697261.203603:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697261.203612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697261/real 1787697261] req@ffff9d85b9d2c000 x1874531134389248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697261.203624:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134389248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697261.203631:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697261.203634:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697261.203637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134389248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697261.203642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134389376:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697261.203659:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697261.203669:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697261/real 1787697261] req@ffff9d85b9d2ca80 x1874531134389376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697261.203694:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134389376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697261.203702:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697261.203705:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697261.203709:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134389376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697261.203727:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134389504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697261.203746:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697261.203757:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697261/real 1787697261] req@ffff9d85b9d2f480 x1874531134389504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697261.203769:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134389504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697261.203777:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697261.203781:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697261.203784:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134389504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697261.203790:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134389632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697261.203804:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697261.203813:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697261/real 1787697261] req@ffff9d85b9d2fb80 x1874531134389632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697261.203823:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134389632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697261.203837:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697261.203841:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697261.203843:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134389632:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697261.779073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697262.803070:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697263.827080:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697264.851050:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697265.875093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697266.322960:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697266.322970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697266.322975:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697266.322985:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5168 00000100:00080000:1.0:1787697266.322996:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697266.323003:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697266.323050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134389760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697266.323078:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697266.323082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697266.323084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697266.323088:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5168 00000100:00080000:1.0:1787697266.323092:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697266.323096:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697266.323108:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697266.323122:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134389888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697266.323143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697266.323144:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134389760:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697266.323146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697266.323150:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697266.323161:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5168 00000100:00080000:1.0:1787697266.323167:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697266.323170:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697266.323175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697266.323187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134390016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697266.323197:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697266.323201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697266.323202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697266.323206:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5168 00000100:00080000:1.0:1787697266.323209:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697266.323211:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697266.323216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697266.323224:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134390144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697266.323230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697266.323236:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787697266.323237:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697266.323241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697266.323247:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5168 00000100:00080000:1.0:1787697266.323251:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697266.323254:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697266.323256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697266.323262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134390272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697266.323270:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697266.323273:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697266.323275:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697266.323278:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5168 00000100:00080000:1.0:1787697266.323280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697266.323283:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697266.323285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697266.323292:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134390400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697266.323297:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697266.323299:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697266.323301:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697266.323309:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5168 00000100:00100000:3.0:1787697266.323311:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697266/real 1787697266] req@ffff9d85b9d2c380 x1874531134389760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697266.323313:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697266.323324:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697266.323337:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697266.323350:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134390528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697266.323365:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134389760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697266.323378:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697266.323384:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697266.323388:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134389760:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697266.323412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134389888:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697266.323437:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697266.323449:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697266/real 1787697266] req@ffff9d85b9d2c700 x1874531134389888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697266.323457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134389888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697266.323466:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697266.323468:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697266.323470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134389888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697266.323474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697266.323496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697266.323503:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697266/real 1787697266] req@ffff9d85b9d2ed80 x1874531134390016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697266.323510:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134390016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697266.323514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697266.323516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697266.323518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697266.323521:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697266.323531:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697266.323536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697266/real 1787697266] req@ffff9d85b9d2d500 x1874531134390144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697266.323542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134390144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697266.323547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697266.323548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697266.323551:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697266.323554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134390272:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697266.323562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697266.323567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697266/real 1787697266] req@ffff9d85b9d2e680 x1874531134390272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697266.323572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134390272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697266.323577:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697266.323578:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697266.323580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134390272:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697266.323583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134390400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697266.323608:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697266.323615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697266/real 1787697266] req@ffff9d85b9d2dc00 x1874531134390400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697266.323622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134390400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697266.323626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697266.323628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697266.323630:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134390400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697266.323634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134390528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697266.323644:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697266.323649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697266/real 1787697266] req@ffff9d85b9d2f800 x1874531134390528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697266.323655:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134390528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697266.323660:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697266.323662:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697266.323663:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134390528:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697266.899100:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697267.923105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697268.947028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697269.971087:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697270.995077:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697271.442980:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697271.442990:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697271.442994:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697271.443005:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5173 00000100:00080000:1.0:1787697271.443016:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697271.443023:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697271.443065:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134390656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697271.443090:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697271.443092:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697271.443094:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697271.443098:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5173 00000100:00080000:1.0:1787697271.443101:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697271.443105:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697271.443116:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697271.443129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134390784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697271.443139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697271.443143:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697271.443146:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697271.443149:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5173 00000100:00080000:1.0:1787697271.443153:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697271.443157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697271.443163:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697271.443179:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134390912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697271.443180:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134390656:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697271.443204:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697271.443207:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697271.443210:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697271.443214:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5173 00000100:00080000:1.0:1787697271.443218:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697271.443221:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697271.443226:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697271.443236:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134391040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697271.443244:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697271.443246:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697271.443247:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697271.443249:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5173 00000100:00080000:1.0:1787697271.443261:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697271.443265:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697271.443274:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697271.443275:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697271.443287:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134391168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697271.443295:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697271.443298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697271.443300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697271.443304:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5173 00000100:00080000:1.0:1787697271.443306:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697271.443308:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697271.443311:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697271.443319:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134391296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697271.443324:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697271.443326:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697271.443328:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697271.443334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697271/real 1787697271] req@ffff9d85b9d2f800 x1874531134390656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697271.443335:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5173 00000100:00080000:1.0:1787697271.443341:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697271.443350:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697271.443357:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697271.443375:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134391424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697271.443382:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134390656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697271.443421:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697271.443427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697271.443431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134390656:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697271.443441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390784:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697271.443464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697271.443475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697271/real 1787697271] req@ffff9d85b9d2dc00 x1874531134390784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697271.443491:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134390784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697271.443498:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697271.443518:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697271.443521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390784:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697271.443545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697271.443562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697271.443572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697271/real 1787697271] req@ffff9d85b9d2e680 x1874531134390912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697271.443582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134390912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697271.443588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697271.443591:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697271.443594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134390912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697271.443600:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697271.443617:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697271.443626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697271/real 1787697271] req@ffff9d85b9d2d500 x1874531134391040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697271.443638:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134391040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697271.443650:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697271.443653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697271.443656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697271.443662:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134391168:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697271.443676:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697271.443685:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697271/real 1787697271] req@ffff9d85b9d2ed80 x1874531134391168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697271.443695:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134391168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697271.443703:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697271.443706:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697271.443709:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134391168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697271.443715:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134391296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697271.443731:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697271.443740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697271/real 1787697271] req@ffff9d85b9d2c700 x1874531134391296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697271.443749:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134391296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697271.443755:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697271.443758:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697271.443761:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134391296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697271.443767:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134391424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697271.443782:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697271.443791:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697271/real 1787697271] req@ffff9d85b9d2c380 x1874531134391424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697271.443802:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134391424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697271.443809:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697271.443811:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697271.443814:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134391424:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697272.019060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697273.043122:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697274.067044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697275.091118:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697276.114976:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697276.563013:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697276.563024:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697276.563028:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697276.563040:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5178 00000100:00080000:1.0:1787697276.563051:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697276.563058:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697276.563113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134391552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697276.563156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697276.563161:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697276.563163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697276.563168:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5178 00000100:00080000:1.0:1787697276.563171:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697276.563175:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697276.563187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697276.563201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134391680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697276.563213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697276.563216:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134391552:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697276.563217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697276.563222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697276.563227:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5178 00000100:00080000:1.0:1787697276.563230:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697276.563233:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697276.563238:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697276.563248:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134391808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697276.563255:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697276.563257:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697276.563258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697276.563261:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5178 00000100:00080000:1.0:1787697276.563265:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697276.563268:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697276.563273:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697276.563285:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134391936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697276.563288:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697276.563298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697276.563302:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697276.563304:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697276.563309:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5178 00000100:00080000:1.0:1787697276.563325:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697276.563330:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697276.563336:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697276.563349:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134392064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697276.563356:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697276/real 1787697276] req@ffff9d85b9d2fb80 x1874531134391552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697276.563361:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697276.563368:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697276.563371:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697276.563377:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5178 00000100:00080000:1.0:1787697276.563382:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697276.563387:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134391552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697276.563389:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697276.563397:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697276.563405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697276.563421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697276.563424:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134392192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697276.563428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134391552:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697276.563451:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697276.563455:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697276.563457:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697276.563461:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5178 00000100:00100000:3.0:1787697276.563467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391680:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697276.563471:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697276.563478:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697276.563484:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697276.563497:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134392320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697276.563506:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697276.563525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697276/real 1787697276] req@ffff9d85b9d2f480 x1874531134391680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697276.563539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134391680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697276.563545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697276.563548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697276.563550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697276.563554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697276.563569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697276.563576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697276/real 1787697276] req@ffff9d85b9d2ca80 x1874531134391808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697276.563583:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134391808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697276.563588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697276.563589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697276.563591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697276.563594:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697276.563605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697276.563610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697276/real 1787697276] req@ffff9d85b9d2c000 x1874531134391936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697276.563617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134391936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697276.563622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697276.563624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697276.563625:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134391936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697276.563628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392064:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697276.563637:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697276.563642:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697276/real 1787697276] req@ffff9d85b9d2ce00 x1874531134392064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697276.563649:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134392064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697276.563653:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697276.563655:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697276.563657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697276.563659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697276.563668:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697276.563672:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697276/real 1787697276] req@ffff9d85b9d2e300 x1874531134392192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697276.563678:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134392192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697276.563687:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697276.563689:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697276.563690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697276.563705:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697276.563714:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697276.563719:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697276/real 1787697276] req@ffff9d85b9d2fb80 x1874531134392320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697276.563726:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134392320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697276.563731:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697276.563732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697276.563734:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392320:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697277.139094:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697278.163069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697279.187006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697280.211015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697281.235057:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697281.683003:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697281.683015:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697281.683019:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697281.683031:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5183 00000100:00080000:0.0:1787697281.683044:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697281.683053:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697281.683101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134392448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697281.683132:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697281.683136:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697281.683137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697281.683141:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5183 00000100:00080000:0.0:1787697281.683144:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697281.683147:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697281.683157:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697281.683169:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134392576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697281.683175:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134392448:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697281.683180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697281.683185:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697281.683187:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697281.683191:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5183 00000100:00080000:0.0:1787697281.683194:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697281.683198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697281.683203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697281.683213:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134392704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697281.683221:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697281.683224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697281.683227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697281.683230:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5183 00000100:00080000:0.0:1787697281.683233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697281.683237:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697281.683242:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697281.683253:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697281.683254:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134392832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697281.683265:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697281.683269:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697281.683271:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697281.683276:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5183 00000100:00080000:0.0:1787697281.683279:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697281.683283:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697281.683288:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697281.683299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134392960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697281.683307:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697281.683309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697281/real 1787697281] req@ffff9d85b9d0a300 x1874531134392448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697281.683315:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697281.683320:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697281.683339:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5183 00000100:00080000:0.0:1787697281.683343:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697281.683347:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697281.683353:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697281.683354:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134392448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697281.683368:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134393088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697281.683370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697281.683378:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697281.683381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134392448:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697281.683389:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697281.683395:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697281.683397:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697281.683402:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5183 00000100:00100000:3.0:1787697281.683405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134392576:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697281.683407:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697281.683415:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697281.683420:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697281.683428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697281.683436:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134393216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697281.683440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697281/real 1787697281] req@ffff9d85b9d09180 x1874531134392576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697281.683462:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134392576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697281.683468:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697281.683470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697281.683472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134392576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697281.683476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134392704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697281.683499:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697281.683505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697281/real 1787697281] req@ffff9d85b9d09500 x1874531134392704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697281.683513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134392704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697281.683517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697281.683519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697281.683520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134392704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697281.683524:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134392832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697281.683534:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697281.683539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697281/real 1787697281] req@ffff9d85b9d08700 x1874531134392832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697281.683546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134392832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697281.683550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697281.683552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697281.683553:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134392832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697281.683556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392960:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697281.683566:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697281.683571:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697281/real 1787697281] req@ffff9d85b9d0b800 x1874531134392960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697281.683577:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134392960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697281.683581:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697281.683583:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697281.683584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134392960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697281.683587:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697281.683596:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697281.683600:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697281/real 1787697281] req@ffff9d85b9d09c00 x1874531134393088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697281.683607:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134393088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697281.683612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697281.683613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697281.683615:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697281.683625:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697281.683634:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697281.683639:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697281/real 1787697281] req@ffff9d85b9d09f80 x1874531134393216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697281.683645:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134393216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697281.683649:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697281.683651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697281.683652:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393216:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697282.259048:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697283.283032:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697284.307005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697285.330996:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697286.354975:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697286.802929:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697286.802939:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697286.802943:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697286.802953:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5188 00000100:00080000:0.0:1787697286.802963:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697286.802971:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697286.803005:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134393344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697286.803032:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697286.803035:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697286.803037:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697286.803041:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5188 00000100:00080000:0.0:1787697286.803044:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697286.803048:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697286.803056:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697286.803069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134393472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697286.803073:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134393344:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697286.803086:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697286.803090:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697286.803092:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697286.803096:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5188 00000100:00080000:0.0:1787697286.803100:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697286.803104:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697286.803110:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697286.803121:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134393600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697286.803129:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697286.803132:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697286.803136:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697286.803138:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697286.803143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5188 00000100:00080000:0.0:1787697286.803148:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697286.803151:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697286.803156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697286.803165:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697286/real 1787697286] req@ffff9d85b9d0b480 x1874531134393344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697286.803169:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134393728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697286.803201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697286.803204:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697286.803205:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697286.803209:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5188 00000100:00080000:0.0:1787697286.803212:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697286.803213:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134393344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697286.803221:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:3.0:1787697286.803226:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787697286.803228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697286.803232:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697286.803254:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134393856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697286.803256:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134393344:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697286.803269:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697286.803272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697286.803274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697286.803278:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5188 00000100:00100000:3.0:1787697286.803281:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134393472:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697286.803283:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697286.803290:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697286.803294:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697286.803303:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697286.803307:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134393984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697286.803315:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697286/real 1787697286] req@ffff9d85b9d0a680 x1874531134393472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697286.803334:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697286.803338:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134393472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697286.803341:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697286.803344:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697286.803352:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5188 00000100:00080000:3.0:1787697286.803352:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697286.803359:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697286.803361:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787697286.803365:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697286.803370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134393472:192.168.201.137@tcp:38:kworker.0 00000008:00100000:0.0:1787697286.803373:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697286.803379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134393600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697286.803399:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134394112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697286.803404:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697286.803422:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697286/real 1787697286] req@ffff9d85b9d0bb80 x1874531134393600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697286.803436:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134393600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697286.803449:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697286.803453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697286.803456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134393600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697286.803462:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134393728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697286.803481:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697286.803491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697286/real 1787697286] req@ffff9d85b9d0ad80 x1874531134393728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697286.803497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134393728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697286.803502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697286.803504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697286.803506:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134393728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697286.803509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393856:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697286.803520:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697286.803525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697286/real 1787697286] req@ffff9d85b9d08000 x1874531134393856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697286.803531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134393856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697286.803535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697286.803537:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697286.803539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697286.803550:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697286.803559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697286.803564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697286/real 1787697286] req@ffff9d85b9d08a80 x1874531134393984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697286.803570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134393984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697286.803575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697286.803576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697286.803585:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134393984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697286.803588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134394112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697286.803600:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697286.803605:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697286/real 1787697286] req@ffff9d85b9d08e00 x1874531134394112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697286.803616:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134394112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697286.803623:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697286.803626:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697286.803629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134394112:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697287.379011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697288.403017:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697289.426995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697290.451006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697291.475001:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697291.922961:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697291.922971:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697291.922975:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697291.922984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5193 00000100:00080000:3.0:1787697291.922996:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697291.923003:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697291.923032:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134394240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697291.923053:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697291.923056:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697291.923057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697291.923060:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5193 00000100:00080000:3.0:1787697291.923062:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697291.923064:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697291.923071:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697291.923079:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134394368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697291.923086:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697291.923088:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697291.923089:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697291.923091:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5193 00000100:00080000:3.0:1787697291.923093:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697291.923097:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134394240:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697291.923098:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697291.923106:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697291.923113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134394496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697291.923119:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697291.923121:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697291.923129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697291.923132:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5193 00000100:00080000:3.0:1787697291.923135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697291.923138:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697291.923140:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697291.923159:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134394624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697291.923165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697291.923167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697291.923168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697291.923171:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697291.923175:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5193 00000100:00080000:3.0:1787697291.923178:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697291.923181:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697291.923185:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697291.923195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134394752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697291.923203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697291.923206:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697291.923209:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697291.923213:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697291/real 1787697291] req@ffff9d85c0fb2300 x1874531134394240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697291.923217:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5193 00000100:00080000:3.0:1787697291.923225:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697291.923232:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697291.923234:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134394240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697291.923240:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697291.923243:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697291.923246:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697291.923249:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134394240:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697291.923255:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134394880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697291.923258:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134394368:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697291.923272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697291.923275:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697291.923276:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697291.923282:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697291.923282:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5193 00000100:00080000:3.0:1787697291.923288:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697291.923292:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697291.923296:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697291/real 1787697291] req@ffff9d85c0fb1880 x1874531134394368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697291.923296:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697291.923310:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134395008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697291.923311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134394368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697291.923332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697291.923334:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697291.923336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134394368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697291.923354:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134394496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697291.923365:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697291.923370:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697291/real 1787697291] req@ffff9d85c0fb2a00 x1874531134394496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697291.923377:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134394496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697291.923381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697291.923383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697291.923385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134394496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697291.923388:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134394624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697291.923395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697291.923400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697291/real 1787697291] req@ffff9d85c0fb2680 x1874531134394624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697291.923406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134394624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697291.923414:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697291.923416:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697291.923417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134394624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697291.923420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134394752:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697291.923429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697291.923433:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697291/real 1787697291] req@ffff9d85c0fb3100 x1874531134394752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697291.923439:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134394752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697291.923443:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697291.923445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697291.923446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134394752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697291.923449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134394880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697291.923457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697291.923461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697291/real 1787697291] req@ffff9d85c0fb3480 x1874531134394880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697291.923467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134394880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697291.923471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697291.923472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697291.923474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134394880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697291.923477:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697291.923486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697291.923490:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697291/real 1787697291] req@ffff9d85c0fb1500 x1874531134395008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697291.923496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134395008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697291.923501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697291.923502:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697291.923503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395008:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697292.499006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697293.522985:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697294.547080:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697295.571021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697296.595027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697297.042954:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697297.042965:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697297.042969:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697297.042978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5198 00000100:00080000:3.0:1787697297.042988:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697297.042994:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697297.043026:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134395136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697297.043050:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697297.043053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697297.043054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697297.043056:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5198 00000100:00080000:3.0:1787697297.043080:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697297.043084:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697297.043091:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697297.043101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134395264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697297.043108:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697297.043112:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134395136:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697297.043112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697297.043114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697297.043124:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5198 00000100:00080000:3.0:1787697297.043127:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697297.043131:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697297.043136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697297.043146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134395392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697297.043156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697297.043159:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697297.043161:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697297.043165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5198 00000100:00080000:3.0:1787697297.043169:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697297.043172:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787697297.043176:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697297.043177:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697297.043190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134395520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697297.043196:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697297.043198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697297.043200:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697297.043202:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5198 00000100:00080000:3.0:1787697297.043204:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697297.043207:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697297.043210:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697297.043215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134395648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697297.043220:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697297.043223:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697297.043224:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697297.043226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5198 00000100:00080000:3.0:1787697297.043228:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697297.043233:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697297.043235:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697297/real 1787697297] req@ffff9d85c0fb1f80 x1874531134395136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697297.043236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697297.043257:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134395136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697297.043263:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134395776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697297.043277:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697297.043280:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697297.043285:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697297.043286:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697297.043291:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697297.043293:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134395136:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697297.043297:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5198 00000100:00080000:3.0:1787697297.043303:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697297.043307:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697297.043312:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697297.043323:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134395904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697297.043325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134395264:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697297.043356:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697297.043371:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697297/real 1787697297] req@ffff9d85c0fb0000 x1874531134395264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697297.043383:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134395264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697297.043392:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697297.043395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697297.043398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134395264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697297.043404:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134395392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697297.043423:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697297.043434:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697297/real 1787697297] req@ffff9d85c0fb0700 x1874531134395392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697297.043456:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134395392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697297.043463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697297.043467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697297.043470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134395392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697297.043475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134395520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697297.043490:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697297.043500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697297/real 1787697297] req@ffff9d85c0fb1500 x1874531134395520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697297.043520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134395520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697297.043529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697297.043533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697297.043536:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134395520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697297.043542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395648:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697297.043560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697297.043579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697297/real 1787697297] req@ffff9d85c0fb3480 x1874531134395648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697297.043591:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134395648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697297.043599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697297.043602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697297.043605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697297.043610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697297.043624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697297.043632:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697297/real 1787697297] req@ffff9d85c0fb3100 x1874531134395776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697297.043642:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134395776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697297.043649:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697297.043651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697297.043654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697297.043669:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697297.043682:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697297.043697:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697297/real 1787697297] req@ffff9d85c0fb2680 x1874531134395904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697297.043708:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134395904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697297.043715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697297.043718:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697297.043721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134395904:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697297.619010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697298.642993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697299.667034:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697300.691005:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697301.715014:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697302.162919:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697302.162930:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697302.162934:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697302.162950:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5203 00000100:00080000:3.0:1787697302.162960:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697302.162967:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697302.163013:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134396032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697302.163039:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697302.163043:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697302.163045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697302.163048:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5203 00000100:00080000:3.0:1787697302.163052:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697302.163056:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697302.163066:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697302.163078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134396032:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697302.163082:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134396160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697302.163095:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697302.163098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697302.163100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697302.163104:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5203 00000100:00080000:3.0:1787697302.163108:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697302.163112:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697302.163118:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697302.163129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134396288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697302.163139:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697302.163141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697302.163145:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697302.163147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697302.163158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5203 00000100:00080000:3.0:1787697302.163162:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697302.163166:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697302.163171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697302.163185:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134396416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697302.163187:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697302/real 1787697302] req@ffff9d85c0fb2a00 x1874531134396032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697302.163208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697302.163213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697302.163215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697302.163219:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134396032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697302.163221:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5203 00000100:00080000:3.0:1787697302.163227:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697302.163235:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697302.163236:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697302.163243:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787697302.163243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697302.163250:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134396032:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697302.163258:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134396544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697302.163267:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697302.163270:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697302.163271:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697302.163276:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134396160:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697302.163276:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5203 00000100:00080000:3.0:1787697302.163280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697302.163285:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697302.163289:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697302.163298:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134396672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697302.163304:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697302.163307:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787697302.163310:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697302.163311:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697302.163317:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5203 00000100:00080000:3.0:1787697302.163320:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697302.163323:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697302.163326:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697302.163332:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697302/real 1787697302] req@ffff9d85c0fb1880 x1874531134396160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697302.163334:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134396800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697302.163353:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134396160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697302.163366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697302.163369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697302.163374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134396160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697302.163381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134396288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697302.163400:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697302.163411:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697302/real 1787697302] req@ffff9d85c0fb2300 x1874531134396288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697302.163422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134396288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697302.163429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697302.163432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697302.163435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134396288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697302.163441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134396416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697302.163459:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697302.163470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697302/real 1787697302] req@ffff9d85c0fb0a80 x1874531134396416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697302.163481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134396416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697302.163489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697302.163492:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697302.163494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134396416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697302.163500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134396544:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697302.163515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697302.163524:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697302/real 1787697302] req@ffff9d85c0fb0e00 x1874531134396544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697302.163534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134396544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697302.163542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697302.163546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697302.163549:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134396544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697302.163565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134396672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697302.163583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697302.163592:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697302/real 1787697302] req@ffff9d85c0fb3800 x1874531134396672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697302.163603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134396672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697302.163610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697302.163613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697302.163616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134396672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697302.163626:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134396800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697302.163640:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697302.163648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697302/real 1787697302] req@ffff9d85c0fb1c00 x1874531134396800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697302.163659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134396800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697302.163666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697302.163668:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697302.163671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134396800:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697302.739054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697303.763071:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697304.787064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697305.811006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697306.835077:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697307.283017:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697307.283026:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697307.283030:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697307.283038:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5209 00000100:00080000:1.0:1787697307.283048:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697307.283054:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697307.283085:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134396928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697307.283107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697307.283109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697307.283110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697307.283113:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5209 00000100:00080000:1.0:1787697307.283115:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697307.283118:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697307.283126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697307.283133:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134397056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697307.283140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697307.283144:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787697307.283145:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134396928:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697307.283146:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697307.283151:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5209 00000100:00080000:1.0:1787697307.283156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697307.283160:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697307.283165:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697307.283176:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134397184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697307.283185:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697307.283189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697307.283191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697307.283194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5209 00000100:00080000:1.0:1787697307.283205:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697307.283209:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:3.0:1787697307.283215:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697307.283219:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697307.283231:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134397312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697307.283238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697307.283240:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697307.283241:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697307.283244:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5209 00000100:00080000:1.0:1787697307.283248:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697307.283252:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697307.283257:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697307.283261:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697307/real 1787697307] req@ffff9d85b9d2c380 x1874531134396928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697307.283279:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134397440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697307.283283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134396928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697307.283302:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697307.283306:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697307.283307:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697307.283307:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697307.283316:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5209 00000100:00080000:3.0:1787697307.283316:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697307.283322:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697307.283326:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134396928:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697307.283329:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697307.283335:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697307.283342:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134397568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697307.283348:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697307.283350:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697307.283352:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697307.283355:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5209 00000100:00080000:1.0:1787697307.283356:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697307.283358:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697307.283362:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697307.283366:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697307.283371:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134397696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697307.283392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697307.283405:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697307/real 1787697307] req@ffff9d85b9d2c700 x1874531134397056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697307.283416:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134397056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697307.283423:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697307.283427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697307.283429:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697307.283435:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697307.283450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697307.283460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697307/real 1787697307] req@ffff9d85b9d2ed80 x1874531134397184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697307.283469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134397184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697307.283476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697307.283478:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697307.283481:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697307.283486:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697307.283504:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697307.283514:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697307/real 1787697307] req@ffff9d85b9d2d500 x1874531134397312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697307.283524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134397312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697307.283532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697307.283534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697307.283547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697307.283552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134397440:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697307.283572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697307.283584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697307/real 1787697307] req@ffff9d85b9d2e680 x1874531134397440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697307.283595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134397440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697307.283602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697307.283605:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697307.283608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134397440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697307.283621:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134397568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697307.283639:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697307.283650:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697307/real 1787697307] req@ffff9d85b9d2dc00 x1874531134397568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697307.283661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134397568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697307.283668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697307.283671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697307.283675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134397568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697307.283693:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134397696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697307.283709:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697307.283718:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697307/real 1787697307] req@ffff9d85b9d2f800 x1874531134397696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697307.283729:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134397696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697307.283736:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697307.283738:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697307.283741:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134397696:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697307.858974:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697308.883006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697309.907035:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697310.931025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697311.955027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697312.402985:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697312.402994:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697312.402997:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697312.403005:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5214 00000100:00080000:2.0:1787697312.403015:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697312.403020:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697312.403049:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134397824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697312.403070:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697312.403073:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697312.403074:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697312.403076:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5214 00000100:00080000:2.0:1787697312.403078:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697312.403080:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697312.403087:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697312.403095:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134397952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697312.403101:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697312.403104:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697312.403105:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697312.403108:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5214 00000100:00080000:2.0:1787697312.403110:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697312.403112:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134397824:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697312.403121:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697312.403127:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697312.403134:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134398080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697312.403141:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697312.403143:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697312.403145:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697312.403149:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5214 00000100:00080000:2.0:1787697312.403162:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697312.403167:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697312.403172:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697312.403183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134398208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697312.403192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697312.403194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697312.403196:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697312.403199:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5214 00000100:00080000:2.0:1787697312.403202:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697312.403204:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397952:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697312.403208:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697312.403220:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697312.403230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134398336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697312.403236:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697312.403239:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697312.403240:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697312.403243:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5214 00000100:00080000:2.0:1787697312.403246:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697312.403251:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697312.403254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697312.403257:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787697312.403261:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134398464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697312.403270:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697312.403272:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697312.403273:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697312.403276:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5214 00000100:00080000:2.0:1787697312.403281:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697312.403284:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697312.403286:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697312.403293:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787697312.403296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134398592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697312.403329:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134398336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697312.403343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134398464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697312.403355:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134398592:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697312.979009:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697314.003017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697315.026993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787697315.475143:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787697315.475185:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697315.475201:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697315.475249:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697315.475257:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697315.475266:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697315.475273:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697315.475280:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697315.475288:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:3.0:1787697315.475358:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697312/real 1787697315] req@ffff9d85babd7480 x1874531134398464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697315.475400:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134398464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697315.475416:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697315.475421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697315.475426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134398464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697315.475436:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697312/real 1787697315] req@ffff9d85babd5c00 x1874531134398592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697315.475449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134398592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697315.475457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697315.475460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697315.475463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134398592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697315.475471:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697312/real 1787697315] req@ffff9d85babd4000 x1874531134398080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697315.475481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134398080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697315.475488:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697315.475492:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697315.475495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697315.475502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697312/real 1787697315] req@ffff9d85babd4380 x1874531134398208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697315.475513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134398208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697315.475521:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697315.475524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697315.475527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697315.475534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697312/real 1787697315] req@ffff9d85babd6300 x1874531134398336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697315.475546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134398336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697315.475553:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697315.475557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697315.475560:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134398336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697315.475567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697312/real 1787697315] req@ffff9d85babd6a00 x1874531134397824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697315.475580:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134397824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697315.475587:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697315.475591:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697315.475594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134397824:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697315.475601:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697312/real 1787697315] req@ffff9d85babd6680 x1874531134397952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697315.475617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134397952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697315.475624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697315.475628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697315.475631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134397952:192.168.201.137@tcp:38:kworker.0 00010000:00010000:2.0:1787697316.051100:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697317.075018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697317.522974:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697317.522984:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697317.522989:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697317.522999:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5219 00000100:00080000:3.0:1787697317.523009:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697317.523016:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697317.523050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134398720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697317.523081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697317.523085:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697317.523087:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697317.523091:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5219 00000100:00080000:3.0:1787697317.523095:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697317.523099:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697317.523109:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697317.523122:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134398848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697317.523132:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134398720:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697317.523160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697317.523164:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697317.523166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697317.523171:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5219 00000100:00080000:3.0:1787697317.523174:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697317.523177:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697317.523192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697317.523203:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697317.523206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134398976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697317.523216:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697317.523220:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697317.523222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697317.523226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5219 00000100:00080000:3.0:1787697317.523229:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697317.523233:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697317.523238:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697317.523250:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697317/real 1787697317] req@ffff9d85c0fb2d80 x1874531134398720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697317.523252:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134399104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697317.523289:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134398720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697317.523290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697317.523298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697317.523303:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697317.523310:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5219 00000100:00080000:1.0:1787697317.523312:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697317.523315:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697317.523320:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697317.523324:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697317.523329:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134398720:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697317.523331:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697317.523347:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134399232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697317.523362:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697317.523364:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398848:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697317.523367:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697317.523370:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697317.523378:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5219 00000100:00080000:3.0:1787697317.523381:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697317.523386:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697317.523391:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697317.523396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697317.523412:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134399360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697317.523423:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697317/real 1787697317] req@ffff9d85c0fb0380 x1874531134398848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697317.523423:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697317.523431:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697317.523439:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697317.523444:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5219 00000100:00100000:1.0:1787697317.523445:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134398848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697317.523450:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697317.523458:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697317.523462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697317.523465:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697317.523469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697317.523474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398848:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697317.523481:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134399488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697317.523484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697317.523515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697317.523529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697317/real 1787697317] req@ffff9d85c0fb3b80 x1874531134398976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697317.523542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134398976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697317.523549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697317.523552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697317.523555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134398976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697317.523562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134399104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697317.523582:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697317.523593:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697317/real 1787697317] req@ffff9d85c0fb1c00 x1874531134399104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697317.523603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134399104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697317.523611:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697317.523615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697317.523618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134399104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697317.523633:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134399232:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697317.523651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697317.523662:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697317/real 1787697317] req@ffff9d85c0fb3800 x1874531134399232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697317.523678:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134399232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697317.523685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697317.523688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697317.523690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134399232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697317.523696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134399360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697317.523724:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697317.523736:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697317/real 1787697317] req@ffff9d85c0fb0e00 x1874531134399360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697317.523747:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134399360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697317.523755:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697317.523758:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697317.523761:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134399360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697317.523766:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134399488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697317.523784:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697317.523794:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697317/real 1787697317] req@ffff9d85c0fb0a80 x1874531134399488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697317.523804:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134399488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697317.523817:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697317.523821:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697317.523823:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134399488:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697318.099025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697319.122999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697320.147017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697321.171011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697322.195069:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697322.642977:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697322.642988:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697322.642992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697322.643003:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5224 00000100:00080000:3.0:1787697322.643015:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697322.643023:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697322.643064:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134399616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697322.643090:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697322.643093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697322.643094:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697322.643097:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5224 00000100:00080000:3.0:1787697322.643099:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697322.643101:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697322.643110:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697322.643118:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134399744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697322.643125:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697322.643127:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697322.643130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697322.643133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5224 00000100:00080000:3.0:1787697322.643135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697322.643138:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697322.643141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697322.643151:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134399616:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697322.643151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134399872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697322.643164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697322.643166:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697322.643167:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697322.643169:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5224 00000100:00080000:3.0:1787697322.643171:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697322.643173:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697322.643176:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697322.643184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134400000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697322.643190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697322.643194:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697322.643196:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697322.643199:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5224 00000100:00080000:3.0:1787697322.643203:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697322.643208:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697322.643213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697322.643225:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134400128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697322.643230:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697322.643239:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697322.643242:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697322.643243:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697322.643247:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5224 00000100:00080000:3.0:1787697322.643250:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697322.643254:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697322.643258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697322.643267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134400256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697322.643273:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697322.643275:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697322.643276:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697322.643278:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5224 00000100:00080000:3.0:1787697322.643281:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697322.643283:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697322.643286:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697322.643288:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697322/real 1787697322] req@ffff9d85c0fb2300 x1874531134399616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697322.643296:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134400384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697322.643309:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134399616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697322.643318:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697322.643322:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697322.643325:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134399616:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697322.643330:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134399744:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697322.643346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697322.643354:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697322/real 1787697322] req@ffff9d85c0fb1880 x1874531134399744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697322.643361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134399744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697322.643366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697322.643368:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697322.643370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134399744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697322.643390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134399872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697322.643401:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697322.643407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697322/real 1787697322] req@ffff9d85c0fb2a00 x1874531134399872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697322.643427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134399872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697322.643435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697322.643438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697322.643441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134399872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697322.643446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697322.643466:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697322.643477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697322/real 1787697322] req@ffff9d85c0fb2680 x1874531134400000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697322.643484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134400000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697322.643489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697322.643491:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697322.643493:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697322.643497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134400128:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697322.643509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697322.643515:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697322/real 1787697322] req@ffff9d85c0fb3100 x1874531134400128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697322.643522:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134400128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697322.643527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697322.643528:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697322.643530:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134400128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697322.643533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134400256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697322.643542:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697322.643547:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697322/real 1787697322] req@ffff9d85c0fb3480 x1874531134400256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697322.643553:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134400256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697322.643558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697322.643559:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697322.643561:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134400256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697322.643563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134400384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697322.643571:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697322.643575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697322/real 1787697322] req@ffff9d85c0fb1500 x1874531134400384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697322.643582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134400384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697322.643590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697322.643592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697322.643594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134400384:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697323.219011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697324.243076:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697325.266986:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697326.291005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697327.315001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697327.763020:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697327.763034:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697327.763039:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697327.763053:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5229 00000100:00080000:3.0:1787697327.763067:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697327.763077:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697327.763127:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134400512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697327.763159:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697327.763163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697327.763165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697327.763169:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5229 00000100:00080000:3.0:1787697327.763173:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697327.763177:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697327.763189:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697327.763203:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134400640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697327.763210:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134400512:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697327.763215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697327.763220:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697327.763222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697327.763226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5229 00000100:00080000:3.0:1787697327.763230:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697327.763234:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697327.763239:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697327.763252:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134400768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697327.763261:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697327.763265:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697327.763267:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697327.763271:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5229 00000100:00080000:3.0:1787697327.763274:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697327.763278:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697327.763283:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697327.763292:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697327.763298:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134400896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697327.763308:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697327.763311:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697327.763313:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697327.763317:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5229 00000100:00080000:3.0:1787697327.763320:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697327.763324:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697327.763328:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697327.763338:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134401024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697327.763345:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697327/real 1787697327] req@ffff9d85c0fb0700 x1874531134400512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697327.763349:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697327.763359:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697327.763362:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697327.763368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134400512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697327.763369:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5229 00000100:00080000:3.0:1787697327.763377:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697327.763388:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697327.763392:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697327.763395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697327.763402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134400512:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697327.763402:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697327.763419:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134401152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697327.763428:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697327.763432:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697327.763436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400640:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697327.763436:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697327.763445:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5229 00000100:00080000:3.0:1787697327.763449:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697327.763453:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697327.763458:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697327.763467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697327.763472:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134401280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697327.763482:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697327/real 1787697327] req@ffff9d85c0fb0000 x1874531134400640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697327.763495:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134400640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697327.763503:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697327.763507:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697327.763510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697327.763516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697327.763548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697327.763559:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697327/real 1787697327] req@ffff9d85c0fb1f80 x1874531134400768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697327.763569:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134400768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697327.763576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697327.763579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697327.763582:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697327.763587:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697327.763602:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697327.763610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697327/real 1787697327] req@ffff9d85c0fb1500 x1874531134400896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697327.763620:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134400896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697327.763628:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697327.763631:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697327.763634:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134400896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697327.763639:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401024:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697327.763657:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697327.763667:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697327/real 1787697327] req@ffff9d85c0fb3480 x1874531134401024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697327.763678:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134401024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697327.763685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697327.763689:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697327.763691:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697327.763698:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697327.763717:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697327.763727:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697327/real 1787697327] req@ffff9d85c0fb3100 x1874531134401152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697327.763734:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134401152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697327.763739:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697327.763741:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697327.763743:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697327.763756:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697327.763767:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697327.763772:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697327/real 1787697327] req@ffff9d85c0fb2680 x1874531134401280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697327.763779:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134401280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697327.763788:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697327.763789:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697327.763791:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401280:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697328.339045:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697329.363049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697330.387068:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697331.411052:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697332.434976:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697332.882902:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697332.882911:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697332.882915:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697332.882925:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5234 00000100:00080000:3.0:1787697332.882936:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697332.882942:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697332.882979:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134401408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697332.883004:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697332.883008:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697332.883010:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697332.883015:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5234 00000100:00080000:3.0:1787697332.883019:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697332.883023:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697332.883048:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134401408:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697332.883060:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697332.883073:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134401536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697332.883090:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697332.883093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697332.883095:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697332.883111:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5234 00000100:00080000:3.0:1787697332.883115:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697332.883120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697332.883126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697332.883151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134401664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697332.883155:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697332.883166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697332.883170:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697332.883172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697332.883176:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5234 00000100:00080000:3.0:1787697332.883180:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697332.883182:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697332.883186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697332.883194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134401792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697332.883201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697332.883204:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697332.883206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697332.883209:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5234 00000100:00080000:3.0:1787697332.883212:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697332.883219:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697332.883220:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697332/real 1787697332] req@ffff9d85c0fb2a00 x1874531134401408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697332.883225:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697332.883242:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134401920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697332.883247:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134401408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697332.883252:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697332.883255:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697332.883258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697332.883264:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5234 00000100:00080000:3.0:1787697332.883269:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697332.883275:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697332.883278:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697332.883279:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697332.883287:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697332.883291:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134402048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697332.883296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134401408:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697332.883321:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697332.883322:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134401536:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697332.883328:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697332.883332:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697332.883336:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5234 00000100:00080000:3.0:1787697332.883340:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697332.883345:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697332.883351:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697332.883360:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697332.883364:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134402176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697332.883380:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697332/real 1787697332] req@ffff9d85c0fb1880 x1874531134401536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697332.883393:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134401536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697332.883401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697332.883405:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697332.883408:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134401536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697332.883414:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134401664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697332.883433:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697332.883457:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697332/real 1787697332] req@ffff9d85c0fb2300 x1874531134401664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697332.883469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134401664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697332.883476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697332.883480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697332.883483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134401664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697332.883489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134401792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697332.883507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697332.883518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697332/real 1787697332] req@ffff9d85c0fb0a80 x1874531134401792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697332.883528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134401792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697332.883536:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697332.883539:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697332.883542:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134401792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697332.883547:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401920:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697332.883563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697332.883579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697332/real 1787697332] req@ffff9d85c0fb0e00 x1874531134401920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697332.883590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134401920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697332.883598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697332.883601:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697332.883604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134401920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697332.883621:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697332.883639:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697332.883650:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697332/real 1787697332] req@ffff9d85c0fb3800 x1874531134402048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697332.883660:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134402048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697332.883667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697332.883670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697332.883673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697332.883678:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697332.883692:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697332.883702:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697332/real 1787697332] req@ffff9d85c0fb1c00 x1874531134402176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697332.883712:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134402176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697332.883739:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697332.883742:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697332.883745:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402176:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697333.459047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697334.482989:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697335.506996:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697336.531016:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697337.555043:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697338.002968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697338.002978:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697338.002983:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697338.002992:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5239 00000100:00080000:3.0:1787697338.003002:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697338.003010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697338.003044:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134402304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697338.003070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697338.003073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697338.003075:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697338.003080:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5239 00000100:00080000:3.0:1787697338.003084:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697338.003088:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697338.003097:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697338.003107:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134402304:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697338.003110:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134402432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697338.003122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697338.003125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697338.003127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697338.003131:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5239 00000100:00080000:3.0:1787697338.003134:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697338.003138:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697338.003144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697338.003153:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134402560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697338.003161:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697338.003168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697338.003172:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697338.003174:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697338.003179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5239 00000100:00080000:3.0:1787697338.003183:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697338.003186:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697338.003191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697338.003202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134402688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697338.003207:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697338/real 1787697338] req@ffff9d85c0fb3b80 x1874531134402304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697338.003220:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697338.003224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697338.003227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697338.003231:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134402304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697338.003233:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5239 00000100:00080000:3.0:1787697338.003240:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697338.003244:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697338.003248:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697338.003250:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697338.003257:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134402304:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697338.003258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697338.003272:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134402816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697338.003275:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134402432:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697338.003290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697338.003294:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697338.003296:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697338.003301:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5239 00000800:00000100:1.0:1787697338.003306:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697338.003307:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697338.003313:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697338.003318:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697338.003323:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697338/real 1787697338] req@ffff9d85c0fb0380 x1874531134402432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697338.003332:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134402944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697338.003343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134402432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697338.003344:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697338.003365:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697338.003367:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697338.003372:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5239 00000100:00080000:3.0:1787697338.003375:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697338.003383:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697338.003386:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697338.003389:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697338.003392:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697338.003396:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134402432:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697338.003404:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134402560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697338.003405:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134403072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697338.003437:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697338.003451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697338/real 1787697338] req@ffff9d85c0fb2d80 x1874531134402560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697338.003463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134402560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697338.003470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697338.003472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697338.003474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134402560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697338.003479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134402688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697338.003493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697338.003499:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697338/real 1787697338] req@ffff9d85c0fb1c00 x1874531134402688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697338.003506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134402688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697338.003511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697338.003513:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697338.003514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134402688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697338.003524:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402816:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697338.003548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697338.003560:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697338/real 1787697338] req@ffff9d85c0fb3800 x1874531134402816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697338.003571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134402816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697338.003579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697338.003581:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697338.003584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697338.003590:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697338.003609:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697338.003619:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697338/real 1787697338] req@ffff9d85c0fb0e00 x1874531134402944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697338.003631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134402944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697338.003639:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697338.003642:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697338.003645:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134402944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697338.003651:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697338.003667:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697338.003675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697338/real 1787697338] req@ffff9d85c0fb0a80 x1874531134403072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697338.003681:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134403072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697338.003690:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697338.003692:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697338.003694:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403072:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697338.579021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697339.603063:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697340.626998:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697341.651031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697342.675029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697343.122972:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697343.122979:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697343.122982:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697343.122990:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5244 00000100:00080000:2.0:1787697343.122998:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697343.123003:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697343.123031:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134403200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697343.123050:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697343.123052:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697343.123054:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697343.123056:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5244 00000100:00080000:2.0:1787697343.123058:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697343.123060:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697343.123072:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697343.123086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134403328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697343.123098:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697343.123101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697343.123104:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697343.123110:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134403200:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697343.123112:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5244 00000100:00080000:2.0:1787697343.123121:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697343.123126:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697343.123132:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697343.123142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134403456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697343.123148:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697343.123150:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697343.123151:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697343.123154:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5244 00000100:00080000:2.0:1787697343.123156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697343.123158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697343.123161:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697343.123169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134403584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697343.123174:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697343.123177:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697343.123178:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697343.123180:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5244 00000100:00080000:2.0:1787697343.123182:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697343.123184:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697343.123187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697343.123195:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134403712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697343.123200:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697343.123205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787697343.123209:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697343.123209:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697343.123221:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5244 00000100:00080000:2.0:1787697343.123224:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697343.123226:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697343.123229:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697343.123237:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134403840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697343.123242:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697343.123244:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697343.123246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697343.123249:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5244 00000100:00080000:2.0:1787697343.123250:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697343.123252:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697343.123256:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697343.123273:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134403968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697343.123295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697343/real 1787697343] req@ffff9d85babd7800 x1874531134403200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697343.123317:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134403200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697343.123330:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697343.123334:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697343.123338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134403200:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697343.123367:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134403328:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697343.123392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697343.123418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697343/real 1787697343] req@ffff9d85babd5500 x1874531134403328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697343.123431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134403328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697343.123440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697343.123443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697343.123446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134403328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697343.123454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134403456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697343.123474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697343.123487:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697343/real 1787697343] req@ffff9d85babd6d80 x1874531134403456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697343.123498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134403456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697343.123506:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697343.123510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697343.123512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134403456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697343.123519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134403584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697343.123540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697343.123551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697343/real 1787697343] req@ffff9d85babd6680 x1874531134403584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697343.123561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134403584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697343.123568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697343.123572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697343.123575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134403584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697343.123581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403712:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697343.123598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697343.123609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697343/real 1787697343] req@ffff9d85babd6a00 x1874531134403712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697343.123620:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134403712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697343.123627:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697343.123630:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697343.123633:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697343.123639:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697343.123657:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697343.123667:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697343/real 1787697343] req@ffff9d85babd6300 x1874531134403840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697343.123677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134403840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697343.123684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697343.123687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697343.123690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697343.123696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697343.123712:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697343.123721:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697343/real 1787697343] req@ffff9d85babd4380 x1874531134403968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697343.123731:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134403968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697343.123744:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697343.123747:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697343.123750:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134403968:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697343.699125:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697344.723117:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697345.747186:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697346.770984:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697347.795063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697348.242984:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697348.242995:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697348.243000:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697348.243010:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5250 00000100:00080000:1.0:1787697348.243020:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697348.243028:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697348.243061:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134404096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697348.243087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697348.243091:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697348.243093:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697348.243097:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5250 00000100:00080000:1.0:1787697348.243100:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697348.243104:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697348.243113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697348.243124:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134404096:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697348.243129:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134404224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697348.243141:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697348.243144:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697348.243146:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697348.243150:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5250 00000100:00080000:1.0:1787697348.243154:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697348.243157:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697348.243163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697348.243173:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134404352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697348.243199:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697348.243202:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697348.243204:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697348.243209:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5250 00000800:00000100:3.0:1787697348.243213:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697348.243215:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697348.243221:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697348.243227:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697348.243238:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134404480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697348.243247:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697348.243250:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697348.243252:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697348.243256:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5250 00000100:00080000:1.0:1787697348.243261:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697348.243264:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697348.243269:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697348.243277:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134404608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697348.243286:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697348.243289:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697348.243293:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697348.243294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697348/real 1787697348] req@ffff9d85b9d2f800 x1874531134404096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697348.243298:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5250 00000100:00080000:1.0:1787697348.243302:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697348.243309:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697348.243315:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697348.243324:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134404736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697348.243325:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134404096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697348.243351:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697348.243352:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697348.243357:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697348.243360:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697348.243361:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697348.243366:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5250 00000100:00100000:3.0:1787697348.243369:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134404096:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697348.243371:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697348.243377:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697348.243382:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697348.243390:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134404864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697348.243398:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134404224:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697348.243426:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697348.243440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697348/real 1787697348] req@ffff9d85b9d2dc00 x1874531134404224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697348.243459:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134404224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697348.243467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697348.243470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697348.243472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134404224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697348.243476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134404352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697348.243491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697348.243498:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697348/real 1787697348] req@ffff9d85b9d2e680 x1874531134404352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697348.243505:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134404352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697348.243511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697348.243514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697348.243516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134404352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697348.243519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134404480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697348.243533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697348.243541:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697348/real 1787697348] req@ffff9d85b9d2d500 x1874531134404480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697348.243552:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134404480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697348.243574:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697348.243577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697348.243579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134404480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697348.243582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134404608:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697348.243596:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697348.243602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697348/real 1787697348] req@ffff9d85b9d2ed80 x1874531134404608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697348.243608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134404608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697348.243613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697348.243615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697348.243617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134404608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697348.243620:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134404736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697348.243628:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697348.243646:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697348/real 1787697348] req@ffff9d85b9d2c700 x1874531134404736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697348.243660:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134404736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697348.243667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697348.243670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697348.243674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134404736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697348.243692:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134404864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697348.243711:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697348.243721:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697348/real 1787697348] req@ffff9d85b9d2c380 x1874531134404864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697348.243732:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134404864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697348.243744:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697348.243747:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697348.243749:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134404864:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697348.819085:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697349.843031:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697350.867001:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697351.890974:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697352.915037:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697353.362968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697353.362976:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697353.362979:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697353.362987:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5255 00000100:00080000:3.0:1787697353.362996:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697353.363002:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697353.363033:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134404992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697353.363060:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697353.363064:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697353.363066:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697353.363071:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5255 00000100:00080000:3.0:1787697353.363075:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697353.363078:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697353.363096:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697353.363103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134404992:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697353.363109:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134405120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697353.363116:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697353.363118:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697353.363119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697353.363122:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5255 00000100:00080000:3.0:1787697353.363124:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697353.363126:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697353.363130:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697353.363137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134405248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697353.363148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697353.363151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697353.363152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697353.363156:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697353.363159:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5255 00000100:00080000:3.0:1787697353.363162:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697353.363165:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697353.363169:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697353.363176:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134405376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697353.363182:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697353.363189:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787697353.363193:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697353/real 1787697353] req@ffff9d85c0fb2300 x1874531134404992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.363193:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697353.363196:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5255 00000100:00080000:3.0:1787697353.363201:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697353.363207:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697353.363211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697353.363214:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134404992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.363220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134405504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697353.363225:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.363228:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697353.363231:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697353.363232:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697353.363235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697353.363237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134404992:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697353.363240:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5255 00000100:00080000:3.0:1787697353.363246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697353.363248:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697353.363251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697353.363256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134405632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697353.363262:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134405120:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697353.363264:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697353.363267:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697353.363269:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697353.363272:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5255 00000100:00080000:3.0:1787697353.363275:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697353.363277:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697353.363280:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697353.363281:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697353.363291:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134405760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697353.363295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697353/real 1787697353] req@ffff9d85c0fb1880 x1874531134405120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697353.363307:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134405120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697353.363313:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697353.363316:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697353.363318:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134405120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697353.363323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134405248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697353.363336:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697353.363345:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697353/real 1787697353] req@ffff9d85c0fb2a00 x1874531134405248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697353.363355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134405248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697353.363368:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697353.363371:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697353.363374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134405248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697353.363380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134405376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697353.363398:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697353.363409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697353/real 1787697353] req@ffff9d85c0fb2680 x1874531134405376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697353.363419:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134405376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697353.363427:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697353.363430:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697353.363433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134405376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697353.363439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134405504:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697353.363456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697353.363465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697353/real 1787697353] req@ffff9d85c0fb3100 x1874531134405504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697353.363472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134405504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697353.363476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697353.363478:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697353.363480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134405504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697353.363491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134405632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697353.363502:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697353.363507:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697353/real 1787697353] req@ffff9d85c0fb3480 x1874531134405632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697353.363514:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134405632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697353.363518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697353.363519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697353.363521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134405632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697353.363528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134405760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697353.363538:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697353.363543:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697353/real 1787697353] req@ffff9d85c0fb1500 x1874531134405760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697353.363549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134405760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697353.363553:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697353.363555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697353.363556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134405760:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697353.938983:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697354.963044:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697355.987079:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697357.011120:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697358.035035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697358.482975:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697358.482990:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697358.482993:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697358.483021:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5260 00000100:00080000:1.0:1787697358.483031:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697358.483037:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697358.483067:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134405888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697358.483096:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697358.483100:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697358.483102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697358.483106:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5260 00000100:00080000:1.0:1787697358.483110:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697358.483115:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697358.483124:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697358.483136:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134406016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697358.483140:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134405888:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697358.483154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697358.483157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697358.483159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697358.483163:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5260 00000100:00080000:1.0:1787697358.483167:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697358.483172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697358.483177:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697358.483190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134406144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697358.483205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697358.483207:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697358.483210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697358.483213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697358.483218:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5260 00000100:00080000:1.0:1787697358.483223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697358.483227:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697358.483232:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697358.483256:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134406272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697358.483267:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697358.483270:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697358/real 1787697358] req@ffff9d85b9d2fb80 x1874531134405888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697358.483271:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697358.483275:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697358.483284:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5260 00000100:00080000:1.0:1787697358.483290:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697358.483294:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697358.483299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134405888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697358.483302:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697358.483316:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697358.483320:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134406400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697358.483324:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697358.483335:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697358.483339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134405888:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697358.483340:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697358.483344:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697358.483350:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5260 00000100:00080000:1.0:1787697358.483354:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697358.483358:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697358.483363:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697358.483368:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697358.483375:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134406528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697358.483386:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697358.483389:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697358.483391:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697358.483395:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5260 00000100:00080000:1.0:1787697358.483399:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697358.483402:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697358.483406:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697358.483412:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697358.483422:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697358/real 1787697358] req@ffff9d85b9d2e300 x1874531134406016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697358.483426:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134406656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697358.483453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134406016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697358.483464:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697358.483468:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697358.483472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697358.483479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697358.483497:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697358.483505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697358/real 1787697358] req@ffff9d85b9d2ce00 x1874531134406144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697358.483524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134406144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697358.483531:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697358.483534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697358.483537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697358.483543:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697358.483561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697358.483572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697358/real 1787697358] req@ffff9d85b9d2c000 x1874531134406272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697358.483583:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134406272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697358.483590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697358.483593:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697358.483597:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697358.483602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134406400:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697358.483619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697358.483627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697358/real 1787697358] req@ffff9d85b9d2ca80 x1874531134406400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697358.483634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134406400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697358.483638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697358.483640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697358.483642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134406400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697358.483655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134406528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697358.483666:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697358.483675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697358/real 1787697358] req@ffff9d85b9d2f480 x1874531134406528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697358.483685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134406528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697358.483692:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697358.483695:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697358.483698:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134406528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697358.483703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134406656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697358.483719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697358.483728:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697358/real 1787697358] req@ffff9d85b9d2fb80 x1874531134406656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697358.483738:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134406656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697358.483745:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697358.483748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697358.483751:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134406656:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697359.060047:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697360.083025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697361.107028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697362.131057:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697363.155094:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697363.603054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697363.603066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697363.603071:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697363.603083:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5265 00000100:00080000:3.0:1787697363.603103:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697363.603113:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697363.603162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134406784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697363.603200:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697363.603203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697363.603205:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697363.603210:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5265 00000100:00080000:3.0:1787697363.603214:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697363.603218:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697363.603230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697363.603244:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134406912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697363.603254:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697363.603257:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697363.603259:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697363.603263:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5265 00000100:00080000:3.0:1787697363.603266:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697363.603270:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697363.603275:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697363.603287:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134407040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697363.603295:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697363.603298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697363.603300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697363.603303:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5265 00000100:00080000:3.0:1787697363.603307:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697363.603310:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697363.603315:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697363.603328:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134407168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697363.603336:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697363.603339:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697363.603341:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697363.603345:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5265 00000100:00080000:3.0:1787697363.603348:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697363.603352:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697363.603357:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697363.603367:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134407296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697363.603375:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697363.603378:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697363.603380:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697363.603384:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5265 00000100:00080000:3.0:1787697363.603387:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697363.603390:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697363.603395:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697363.603404:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134407424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697363.603412:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697363.603415:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697363.603417:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697363.603421:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5265 00000100:00080000:3.0:1787697363.603424:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697363.603427:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697363.603431:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697363.603442:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134407552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697363.603541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134406784:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787697363.603627:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697363.603681:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697363/real 1787697363] req@ffff9d85c0fb1f80 x1874531134406784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697363.603701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134406784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697363.603713:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697363.603717:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697363.603721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134406784:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697363.603728:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406912:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697363.603749:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697363.603762:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697363/real 1787697363] req@ffff9d85c0fb0000 x1874531134406912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697363.603773:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134406912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697363.603782:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697363.603785:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697363.603788:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134406912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697363.603794:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697363.603814:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697363.603826:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697363/real 1787697363] req@ffff9d85c0fb0700 x1874531134407040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697363.603837:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134407040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697363.603845:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697363.603848:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697363.603870:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697363.603876:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697363.603903:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697363.603927:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697363/real 1787697363] req@ffff9d85c0fb1500 x1874531134407168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697363.603940:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134407168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697363.603947:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697363.603950:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697363.603953:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697363.603956:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134407296:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697363.603972:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697363.603979:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697363/real 1787697363] req@ffff9d85c0fb3480 x1874531134407296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697363.603986:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134407296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697363.603991:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697363.603993:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697363.603995:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134407296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697363.603998:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134407424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697363.604006:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697363.604011:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697363/real 1787697363] req@ffff9d85c0fb3100 x1874531134407424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697363.604017:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134407424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697363.604026:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697363.604028:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697363.604030:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134407424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697363.604033:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134407552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697363.604041:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697363.604046:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697363/real 1787697363] req@ffff9d85c0fb2680 x1874531134407552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697363.604052:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134407552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697363.604057:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697363.604058:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697363.604060:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134407552:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697364.179038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697365.203006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697366.226997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697367.250997:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697368.275004:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697368.722961:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697368.722972:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697368.722977:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697368.722986:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5270 00000100:00080000:3.0:1787697368.722996:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697368.723004:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697368.723038:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134407680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697368.723066:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697368.723070:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697368.723072:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697368.723076:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5270 00000100:00080000:3.0:1787697368.723080:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697368.723084:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697368.723093:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697368.723106:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134407808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697368.723117:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134407680:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697368.723118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697368.723125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697368.723127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697368.723131:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5270 00000100:00080000:3.0:1787697368.723136:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697368.723139:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697368.723145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697368.723156:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134407936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697368.723165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697368.723169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787697368.723172:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697368.723173:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697368.723179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5270 00000100:00080000:3.0:1787697368.723184:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697368.723189:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697368.723194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697368.723205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134408064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697368.723214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697368.723217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697368.723219:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697368.723225:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697368/real 1787697368] req@ffff9d85c0fb2a00 x1874531134407680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697368.723225:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5270 00000100:00080000:3.0:1787697368.723232:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697368.723240:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697368.723246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697368.723252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134407680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697368.723262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134408192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697368.723265:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697368.723274:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697368.723280:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134407680:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697368.723294:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697368.723298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697368.723300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697368.723305:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5270 00000100:00080000:3.0:1787697368.723308:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697368.723311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407808:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697368.723316:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697368.723323:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697368.723334:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134408320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697368.723344:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697368.723348:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697368.723350:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697368.723354:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697368.723358:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5270 00000100:00080000:3.0:1787697368.723362:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697368.723364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697368/real 1787697368] req@ffff9d85c0fb1880 x1874531134407808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697368.723368:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697368.723375:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697368.723387:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134407808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697368.723389:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134408448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697368.723406:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697368.723411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697368.723416:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697368.723429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697368.723450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697368.723462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697368/real 1787697368] req@ffff9d85c0fb2300 x1874531134407936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697368.723475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134407936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697368.723482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697368.723485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697368.723489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134407936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697368.723494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697368.723511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697368.723529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697368/real 1787697368] req@ffff9d85c0fb0a80 x1874531134408064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697368.723541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134408064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697368.723549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697368.723551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697368.723554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697368.723559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134408192:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697368.723573:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697368.723582:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697368/real 1787697368] req@ffff9d85c0fb0e00 x1874531134408192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697368.723592:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134408192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697368.723598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697368.723600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697368.723602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134408192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697368.723613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134408320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697368.723632:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697368.723638:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697368/real 1787697368] req@ffff9d85c0fb3800 x1874531134408320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697368.723645:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134408320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697368.723653:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697368.723655:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697368.723657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134408320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697368.723660:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134408448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697368.723668:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697368.723673:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697368/real 1787697368] req@ffff9d85c0fb1c00 x1874531134408448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697368.723681:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134408448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697368.723688:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697368.723691:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697368.723693:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134408448:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697369.299095:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697370.323035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697371.346998:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697372.371007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697373.394966:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697373.842957:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697373.842969:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697373.842973:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697373.842983:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5275 00000100:00080000:3.0:1787697373.842994:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697373.843000:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697373.843034:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134408576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697373.843061:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697373.843064:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697373.843066:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697373.843069:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5275 00000100:00080000:3.0:1787697373.843072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697373.843075:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697373.843084:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697373.843095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134408704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697373.843103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697373.843106:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697373.843108:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697373.843113:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134408576:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697373.843114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5275 00000100:00080000:3.0:1787697373.843119:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697373.843123:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697373.843128:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697373.843141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134408832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697373.843155:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697373.843158:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697373.843161:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697373.843164:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5275 00000100:00080000:3.0:1787697373.843167:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697373.843170:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697373.843175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697373.843186:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134408960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697373.843193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697373.843196:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697373.843198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697373.843201:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697373.843205:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5275 00000100:00080000:3.0:1787697373.843209:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697373.843213:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697373.843218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697373.843228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134409088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697373.843236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697373.843239:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697373.843241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697373.843244:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5275 00000100:00080000:3.0:1787697373.843247:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697373.843251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697373.843255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697373.843260:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697373/real 1787697373] req@ffff9d85c0fb2d80 x1874531134408576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697373.843267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134409216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697373.843283:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697373.843286:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697373.843289:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697373.843290:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134408576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697373.843296:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5275 00000100:00080000:3.0:1787697373.843302:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697373.843307:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697373.843308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697373.843312:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697373.843316:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697373.843322:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134408576:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697373.843326:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134409344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697373.843351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408704:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697373.843374:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697373.843387:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697373/real 1787697373] req@ffff9d85c0fb0380 x1874531134408704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697373.843408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134408704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697373.843415:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697373.843419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697373.843422:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697373.843428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697373.843446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697373.843457:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697373/real 1787697373] req@ffff9d85c0fb3b80 x1874531134408832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697373.843476:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134408832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697373.843484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697373.843487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697373.843490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697373.843495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697373.843512:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697373.843523:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697373/real 1787697373] req@ffff9d85c0fb1c00 x1874531134408960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697373.843542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134408960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697373.843550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697373.843553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697373.843556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134408960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697373.843562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697373.843579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697373.843589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697373/real 1787697373] req@ffff9d85c0fb3800 x1874531134409088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697373.843599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134409088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697373.843607:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697373.843609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697373.843622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697373.843628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697373.843646:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697373.843656:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697373/real 1787697373] req@ffff9d85c0fb0e00 x1874531134409216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697373.843667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134409216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697373.843679:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697373.843682:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697373.843684:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697373.843690:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697373.843706:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697373.843716:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697373/real 1787697373] req@ffff9d85c0fb0a80 x1874531134409344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697373.843726:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134409344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697373.843761:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697373.843764:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697373.843768:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409344:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697374.419038:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697375.443010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697376.467040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697377.491013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697378.515052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697378.962978:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697378.962987:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697378.962992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697378.963000:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5280 00000100:00080000:3.0:1787697378.963010:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697378.963017:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697378.963050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134409472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697378.963080:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697378.963083:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697378.963085:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697378.963088:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5280 00000100:00080000:3.0:1787697378.963091:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697378.963093:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697378.963100:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697378.963110:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134409600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697378.963116:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697378.963119:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697378.963120:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697378.963122:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5280 00000100:00080000:3.0:1787697378.963124:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697378.963128:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697378.963129:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134409472:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697378.963132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697378.963144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134409728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697378.963151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697378.963154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697378.963155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697378.963157:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5280 00000100:00080000:3.0:1787697378.963160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697378.963162:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697378.963166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697378.963174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134409856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697378.963179:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697378.963181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697378.963182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697378.963184:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5280 00000100:00080000:3.0:1787697378.963186:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697378.963189:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697378.963191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697378.963197:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134409984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697378.963202:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697378.963204:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697378.963206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697378.963208:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5280 00000100:00080000:3.0:1787697378.963209:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697378.963212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697378.963214:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697378.963222:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134410112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697378.963227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134409600:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697378.963244:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697378.963247:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697378.963249:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697378.963255:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5280 00000100:00080000:3.0:1787697378.963259:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697378.963263:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697378.963269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697378.963273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134409728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697378.963284:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134410240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697378.963304:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134409856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697378.963325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697378.963341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134410112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697378.963362:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134410240:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697379.539029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697380.562996:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697381.586999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697382.035077:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697382.035118:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697382.035134:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697382.035170:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697382.035180:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697382.035188:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697382.035195:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697382.035202:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697382.035209:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787697382.035213:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697378/real 1787697382] req@ffff9d85c0fb2a00 x1874531134409728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697382.035236:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134409728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697382.035248:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697382.035254:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697382.035258:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134409728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697382.035266:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697378/real 1787697382] req@ffff9d85c0fb2680 x1874531134409856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697382.035277:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134409856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697382.035285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697382.035288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697382.035291:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134409856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697382.035297:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697378/real 1787697382] req@ffff9d85c0fb3100 x1874531134409984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697382.035308:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134409984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697382.035314:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697382.035318:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697382.035321:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134409984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697382.035327:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697378/real 1787697382] req@ffff9d85c0fb3480 x1874531134410112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697382.035339:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134410112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697382.035346:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697382.035349:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697382.035352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134410112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697382.035357:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697378/real 1787697382] req@ffff9d85c0fb2300 x1874531134409472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697382.035366:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134409472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697382.035372:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697382.035376:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697382.035378:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134409472:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697382.035384:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697378/real 1787697382] req@ffff9d85c0fb1880 x1874531134409600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697382.035394:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134409600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697382.035401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697382.035404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697382.035407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134409600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697382.035435:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697378/real 1787697382] req@ffff9d85c0fb1500 x1874531134410240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697382.035447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134410240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697382.035454:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697382.035457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697382.035460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134410240:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697382.611017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697383.634966:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697384.082973:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697384.082983:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697384.082986:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697384.082996:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5285 00000100:00080000:0.0:1787697384.083005:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697384.083013:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697384.083046:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134410368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697384.083077:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697384.083081:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697384.083083:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697384.083086:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5285 00000100:00080000:0.0:1787697384.083100:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697384.083104:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697384.083112:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697384.083123:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134410496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697384.083129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697384.083132:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697384.083133:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697384.083137:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5285 00000100:00080000:0.0:1787697384.083140:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697384.083143:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697384.083148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697384.083157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134410624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697384.083166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697384.083169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697384.083171:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697384.083174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5285 00000100:00080000:0.0:1787697384.083177:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697384.083180:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697384.083184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697384.083193:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134410752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697384.083201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697384.083203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697384.083205:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697384.083208:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5285 00000100:00080000:0.0:1787697384.083211:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697384.083216:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697384.083219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697384.083226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134410880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697384.083235:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697384.083238:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697384.083240:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697384.083243:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5285 00000100:00080000:0.0:1787697384.083246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697384.083250:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697384.083254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697384.083264:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134411008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697384.083273:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697384.083276:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697384.083278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697384.083281:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5285 00000100:00080000:0.0:1787697384.083294:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697384.083298:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697384.083302:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697384.083311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134411136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697384.083421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134410368:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787697384.083486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697384.083529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697384/real 1787697384] req@ffff9d85b9d08e00 x1874531134410368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697384.083547:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134410368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697384.083569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697384.083573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697384.083577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134410368:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697384.083585:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134410496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697384.083607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697384.083627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697384/real 1787697384] req@ffff9d85b9d08a80 x1874531134410496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697384.083639:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134410496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697384.083648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697384.083651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697384.083654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134410496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697384.083660:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134410624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697384.083679:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697384.083689:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697384/real 1787697384] req@ffff9d85b9d08000 x1874531134410624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697384.083699:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134410624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697384.083705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697384.083708:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697384.083710:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134410624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697384.083715:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134410752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697384.083732:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697384.083740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697384/real 1787697384] req@ffff9d85b9d0ad80 x1874531134410752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697384.083750:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134410752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697384.083761:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697384.083763:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697384.083766:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134410752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697384.083772:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134410880:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697384.083787:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697384.083797:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697384/real 1787697384] req@ffff9d85b9d0bb80 x1874531134410880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697384.083806:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134410880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697384.083813:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697384.083815:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697384.083818:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134410880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697384.083824:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697384.083840:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697384.083882:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697384/real 1787697384] req@ffff9d85b9d0a680 x1874531134411008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697384.083895:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134411008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697384.083902:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697384.083906:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697384.083909:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697384.083915:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697384.083934:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697384.083944:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697384/real 1787697384] req@ffff9d85b9d0b480 x1874531134411136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697384.083955:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134411136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697384.083963:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697384.083966:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697384.083969:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411136:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697384.659021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697385.682980:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697386.707039:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697387.731065:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697388.755077:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697389.202961:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697389.202970:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697389.202974:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697389.202983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5290 00000100:00080000:2.0:1787697389.202993:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697389.203000:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697389.203034:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134411264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697389.203064:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697389.203068:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697389.203069:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697389.203072:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5290 00000100:00080000:2.0:1787697389.203075:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697389.203077:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697389.203084:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697389.203093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134411392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697389.203099:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697389.203101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697389.203102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697389.203105:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5290 00000100:00080000:2.0:1787697389.203107:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697389.203108:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697389.203111:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697389.203119:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134411520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697389.203121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134411264:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697389.203128:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697389.203131:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697389.203132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697389.203134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5290 00000100:00080000:2.0:1787697389.203137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697389.203141:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697389.203145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697389.203155:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134411648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697389.203164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697389.203167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697389.203168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697389.203172:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5290 00000100:00080000:2.0:1787697389.203175:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697389.203179:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697389.203183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697389.203193:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697389.203193:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134411776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697389.203202:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697389.203204:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697389.203206:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697389.203208:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5290 00000100:00080000:2.0:1787697389.203211:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697389.203214:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697389.203216:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697389.203222:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134411904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697389.203232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697389.203234:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697389.203235:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697389.203240:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5290 00000100:00100000:1.0:1787697389.203241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697389/real 1787697389] req@ffff9d85babd4000 x1874531134411264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697389.203242:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697389.203247:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697389.203251:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697389.203261:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134412032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697389.203270:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134411264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697389.203282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697389.203286:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697389.203291:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134411264:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697389.203298:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134411392:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697389.203319:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697389.203332:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697389/real 1787697389] req@ffff9d85babd5c00 x1874531134411392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697389.203343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134411392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697389.203351:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697389.203354:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697389.203357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134411392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697389.203377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134411520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697389.203396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697389.203407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697389/real 1787697389] req@ffff9d85babd7480 x1874531134411520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697389.203417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134411520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697389.203424:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697389.203427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697389.203431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134411520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697389.203437:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134411648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697389.203454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697389.203464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697389/real 1787697389] req@ffff9d85babd5180 x1874531134411648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697389.203489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134411648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697389.203496:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697389.203499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697389.203502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134411648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697389.203507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411776:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697389.203526:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697389.203575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697389/real 1787697389] req@ffff9d85babd4a80 x1874531134411776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697389.203587:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134411776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697389.203594:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697389.203597:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697389.203600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697389.203606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697389.203625:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697389.203636:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697389/real 1787697389] req@ffff9d85babd7b80 x1874531134411904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697389.203646:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134411904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697389.203653:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697389.203656:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697389.203659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134411904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697389.203670:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697389.203689:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697389.203698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697389/real 1787697389] req@ffff9d85babd7100 x1874531134412032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697389.203709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134412032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697389.203717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697389.203720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697389.203723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412032:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697389.779041:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697390.803024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697391.827038:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697392.851458:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697393.875017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697394.322971:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697394.322981:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697394.322984:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697394.322993:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5296 00000100:00080000:3.0:1787697394.323003:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697394.323010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697394.323041:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134412160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697394.323070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697394.323073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697394.323074:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697394.323077:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5296 00000100:00080000:3.0:1787697394.323080:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697394.323082:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697394.323089:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697394.323097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134412288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697394.323104:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697394.323106:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697394.323107:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697394.323110:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5296 00000100:00080000:3.0:1787697394.323112:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697394.323116:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697394.323121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134412160:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697394.323124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697394.323139:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134412416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697394.323149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697394.323152:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697394.323154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697394.323158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5296 00000100:00080000:3.0:1787697394.323162:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697394.323165:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697394.323168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697394.323177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134412544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697394.323185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697394.323188:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697394.323192:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697394.323194:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697394.323199:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5296 00000100:00080000:3.0:1787697394.323203:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697394.323207:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697394.323210:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697394.323219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134412672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697394.323224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697394.323227:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697394.323228:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697394.323231:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5296 00000100:00080000:3.0:1787697394.323234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697394.323235:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697394/real 1787697394] req@ffff9d85c0fb0700 x1874531134412160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697394.323238:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697394.323243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697394.323253:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134412800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697394.323263:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134412160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697394.323265:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697394.323270:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697394.323274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697394.323280:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5296 00000100:00080000:3.0:1787697394.323282:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697394.323285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697394.323285:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697394.323291:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697394.323294:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697394.323302:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134412160:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697394.323302:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134412928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697394.323317:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134412288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697394.323340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697394.323352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697394/real 1787697394] req@ffff9d85c0fb0000 x1874531134412288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697394.323363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134412288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697394.323371:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697394.323373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697394.323377:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134412288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697394.323415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134412416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697394.323434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697394.323444:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697394/real 1787697394] req@ffff9d85c0fb1f80 x1874531134412416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697394.323456:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134412416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697394.323463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697394.323466:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697394.323470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134412416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697394.323475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134412544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697394.323491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697394.323500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697394/real 1787697394] req@ffff9d85c0fb1500 x1874531134412544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697394.323511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134412544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697394.323518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697394.323521:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697394.323525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134412544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697394.323530:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412672:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697394.323544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697394.323552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697394/real 1787697394] req@ffff9d85c0fb1880 x1874531134412672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697394.323562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134412672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697394.323570:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697394.323573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697394.323576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697394.323581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697394.323597:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697394.323607:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697394/real 1787697394] req@ffff9d85c0fb2300 x1874531134412800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697394.323619:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134412800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697394.323626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697394.323629:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697394.323633:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697394.323638:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697394.323655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697394.323665:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697394/real 1787697394] req@ffff9d85c0fb3480 x1874531134412928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697394.323675:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134412928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697394.323681:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697394.323684:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697394.323687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134412928:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697394.899032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697395.923021:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697396.947024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697397.971031:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697398.995025:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697399.443011:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697399.443021:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697399.443025:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697399.443035:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5301 00000100:00080000:1.0:1787697399.443045:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697399.443052:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697399.443086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134413056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697399.443111:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697399.443115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697399.443117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697399.443121:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5301 00000100:00080000:1.0:1787697399.443124:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697399.443128:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697399.443138:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697399.443146:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134413056:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697399.443152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134413184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697399.443164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697399.443167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697399.443169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697399.443171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5301 00000100:00080000:1.0:1787697399.443175:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697399.443178:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697399.443183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697399.443194:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134413312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697399.443202:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697399.443205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697399.443207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697399.443213:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697399.443215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5301 00000100:00080000:1.0:1787697399.443221:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697399.443225:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697399.443230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697399.443241:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134413440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697399.443249:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697399.443252:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697399.443254:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697399.443257:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5301 00000100:00080000:1.0:1787697399.443261:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697399.443268:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443270:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697399/real 1787697399] req@ffff9d85b9d2c380 x1874531134413056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697399.443285:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697399.443303:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134413056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697399.443314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134413568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697399.443318:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697399.443326:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697399.443331:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697399.443333:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134413056:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697399.443338:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697399.443342:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697399.443347:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5301 00000100:00080000:1.0:1787697399.443351:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697399.443361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134413184:192.168.201.137@tcp:38:kworker.0 00000800:00000100:3.0:1787697399.443384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697399/real 1787697399] req@ffff9d85b9d2c700 x1874531134413184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697399.443408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134413184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697399.443426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697399.443430:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697399.443433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134413184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697399.443439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134413312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697399.443461:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443474:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697399/real 1787697399] req@ffff9d85b9d2ed80 x1874531134413312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697399.443475:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134413312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697399.443494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697399.443496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697399.443498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134413312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697399.443501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134413440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697399.443515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697399/real 1787697399] req@ffff9d85b9d2d500 x1874531134413440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697399.443534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134413440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697399.443535:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697399.443546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697399.443549:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697399.443556:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134413696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697399.443557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134413440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697399.443568:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134413568:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697399.443575:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697399.443579:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697399.443581:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697399.443585:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5301 00000100:00080000:1.0:1787697399.443589:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697399.443592:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697399.443598:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697399.443608:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134413824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697399.443695:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443706:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697399/real 1787697399] req@ffff9d85b9d2e680 x1874531134413568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697399.443717:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134413568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697399.443724:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697399.443727:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697399.443730:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134413568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697399.443748:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134413696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697399.443780:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443793:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697399/real 1787697399] req@ffff9d85b9d2dc00 x1874531134413696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697399.443804:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134413696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697399.443812:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697399.443815:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697399.443818:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134413696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697399.443824:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134413824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:3.0:1787697399.443843:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697399.443889:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697399/real 1787697399] req@ffff9d85b9d2f800 x1874531134413824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697399.443902:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134413824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697399.443910:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697399.443913:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697399.443916:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134413824:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697400.019031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697401.043248:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697402.067046:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697403.091029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697404.114993:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697404.563011:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697404.563020:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697404.563024:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697404.563034:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5306 00000100:00080000:3.0:1787697404.563044:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697404.563052:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697404.563088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134413952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697404.563109:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697404.563111:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697404.563112:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697404.563115:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5306 00000100:00080000:3.0:1787697404.563119:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697404.563126:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697404.563135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697404.563149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134414080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697404.563153:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134413952:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697404.563166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697404.563170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697404.563172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697404.563176:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5306 00000100:00080000:3.0:1787697404.563179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697404.563181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697404.563186:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697404.563195:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134414208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697404.563201:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697404.563225:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697404.563227:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697404.563231:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5306 00000100:00080000:3.0:1787697404.563235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697404.563238:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697404.563243:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697404.563249:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697404.563262:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134414336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697404.563275:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697404.563277:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697404.563279:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697404.563282:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5306 00000100:00080000:3.0:1787697404.563285:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697404.563290:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697404.563292:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697404/real 1787697404] req@ffff9d85c0fb3100 x1874531134413952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697404.563294:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697404.563307:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134414464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697404.563315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134413952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697404.563319:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697404.563330:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697404.563333:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697404.563334:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697404.563341:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5306 00000100:00080000:1.0:1787697404.563342:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697404.563349:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697404.563350:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134413952:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697404.563358:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697404.563364:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697404.563376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697404.563378:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134414592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697404.563392:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697404.563395:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697404.563397:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697404.563401:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5306 00000800:00000100:1.0:1787697404.563406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697404.563406:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697404.563413:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697404.563417:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697404.563426:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697404/real 1787697404] req@ffff9d85c0fb2680 x1874531134414080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697404.563428:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134414720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697404.563457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134414080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697404.563467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697404.563470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697404.563474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697404.563480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697404.563500:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697404.563510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697404/real 1787697404] req@ffff9d85c0fb2a00 x1874531134414208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697404.563521:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134414208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697404.563529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697404.563532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697404.563535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697404.563541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697404.563558:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697404.563567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697404/real 1787697404] req@ffff9d85c0fb0a80 x1874531134414336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697404.563578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134414336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697404.563586:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697404.563589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697404.563593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697404.563598:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134414464:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697404.563614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697404.563624:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697404/real 1787697404] req@ffff9d85c0fb0e00 x1874531134414464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697404.563643:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134414464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697404.563650:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697404.563653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697404.563656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134414464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697404.563672:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134414592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697404.563688:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697404.563698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697404/real 1787697404] req@ffff9d85c0fb3800 x1874531134414592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697404.563710:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134414592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697404.563717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697404.563720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697404.563723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134414592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697404.563729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134414720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697404.563744:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697404.563754:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697404/real 1787697404] req@ffff9d85c0fb1c00 x1874531134414720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697404.563765:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134414720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697404.563778:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697404.563781:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697404.563783:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134414720:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697405.139037:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697406.163007:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697407.187017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697408.211031:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697409.235023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697409.682981:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697409.682990:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697409.682994:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697409.683004:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5311 00000100:00080000:2.0:1787697409.683014:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697409.683021:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697409.683059:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134414848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697409.683084:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697409.683088:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697409.683089:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697409.683093:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5311 00000100:00080000:2.0:1787697409.683097:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697409.683100:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697409.683109:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697409.683136:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134414976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697409.683146:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697409.683150:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697409.683152:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697409.683155:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5311 00000100:00080000:2.0:1787697409.683159:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697409.683165:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134414848:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697409.683165:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697409.683172:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697409.683183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134415104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697409.683199:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697409.683203:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697409.683205:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697409.683208:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5311 00000100:00080000:2.0:1787697409.683211:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697409.683214:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697409.683218:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697409.683229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134415232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697409.683234:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697409.683242:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697409.683246:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697409.683247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697409.683252:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5311 00000100:00080000:2.0:1787697409.683255:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697409.683260:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697409.683265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697409.683275:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134415360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697409.683283:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697409.683286:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697409.683289:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697409/real 1787697409] req@ffff9d85babd4e00 x1874531134414848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697409.683290:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697409.683298:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5311 00000100:00080000:2.0:1787697409.683305:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697409.683312:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683317:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134414848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697409.683318:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697409.683331:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134415488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697409.683335:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697409.683338:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697409.683342:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697409.683343:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697409.683346:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697409.683351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134414848:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697409.683352:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5311 00000100:00080000:2.0:1787697409.683357:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697409.683361:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683365:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414976:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787697409.683365:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697409.683376:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134415616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697409.683402:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683415:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697409/real 1787697409] req@ffff9d85babd5f80 x1874531134414976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697409.683427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134414976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697409.683435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697409.683439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697409.683442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134414976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697409.683463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134415104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697409.683482:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683493:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697409/real 1787697409] req@ffff9d85babd4700 x1874531134415104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697409.683504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134415104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697409.683511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697409.683514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697409.683517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134415104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697409.683522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134415232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697409.683538:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683547:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697409/real 1787697409] req@ffff9d85babd7100 x1874531134415232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697409.683557:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134415232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697409.683565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697409.683568:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697409.683570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134415232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697409.683576:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134415360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697409.683590:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697409/real 1787697409] req@ffff9d85babd7b80 x1874531134415360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697409.683609:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134415360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697409.683616:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697409.683619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697409.683622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134415360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697409.683628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134415488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697409.683645:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683655:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697409/real 1787697409] req@ffff9d85babd4a80 x1874531134415488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697409.683666:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134415488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697409.683674:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697409.683677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697409.683680:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134415488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697409.683685:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134415616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697409.683699:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697409.683706:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697409/real 1787697409] req@ffff9d85babd5180 x1874531134415616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697409.683716:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134415616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697409.683729:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697409.683732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697409.683735:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134415616:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697410.259006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697411.283021:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697412.306998:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697413.331046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697414.355010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697414.802973:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697414.802981:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697414.802984:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697414.802991:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5316 00000100:00080000:0.0:1787697414.803001:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697414.803006:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697414.803058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134415744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697414.803076:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697414.803079:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697414.803080:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697414.803083:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5316 00000100:00080000:0.0:1787697414.803085:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697414.803088:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697414.803097:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697414.803105:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134415872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697414.803111:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134415744:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697414.803122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697414.803126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697414.803127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697414.803133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5316 00000100:00080000:0.0:1787697414.803136:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697414.803140:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697414.803146:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697414.803157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134416000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697414.803166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697414.803168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697414.803170:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697414.803173:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5316 00000100:00080000:0.0:1787697414.803177:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697414.803181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697414.803186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697414.803187:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697414.803202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134416128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697414.803211:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697414.803213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697414.803215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697414.803219:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5316 00000100:00080000:0.0:1787697414.803222:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697414.803227:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697414.803230:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697414/real 1787697414] req@ffff9d85b9d09f80 x1874531134415744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697414.803233:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697414.803251:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134416256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697414.803252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134415744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697414.803277:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697414.803279:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697414.803281:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787697414.803286:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697414.803289:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697414.803293:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5316 00000100:00100000:1.0:1787697414.803294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134415744:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697414.803301:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697414.803307:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697414.803312:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697414.803317:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134415872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697414.803326:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134416384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697414.803338:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697414.803338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697414.803344:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697414.803347:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697414.803350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697414/real 1787697414] req@ffff9d85b9d09c00 x1874531134415872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697414.803354:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5316 00000100:00080000:0.0:1787697414.803361:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697414.803366:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697414.803372:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697414.803372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134415872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697414.803382:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697414.803384:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697414.803387:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134416512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697414.803390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134415872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697414.803402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697414.803416:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697414.803422:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697414/real 1787697414] req@ffff9d85b9d0b800 x1874531134416000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697414.803429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134416000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697414.803435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697414.803436:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697414.803438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697414.803441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697414.803463:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697414.803472:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697414/real 1787697414] req@ffff9d85b9d08700 x1874531134416128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697414.803484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134416128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697414.803491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697414.803494:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697414.803497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697414.803501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134416256:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697414.803513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697414.803520:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697414/real 1787697414] req@ffff9d85b9d09500 x1874531134416256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697414.803527:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134416256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697414.803532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697414.803533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697414.803535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134416256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697414.803545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134416384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697414.803554:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697414.803559:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697414/real 1787697414] req@ffff9d85b9d09180 x1874531134416384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697414.803565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134416384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697414.803569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697414.803571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697414.803572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134416384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697414.803575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134416512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697414.803583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697414.803588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697414/real 1787697414] req@ffff9d85b9d0a300 x1874531134416512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697414.803593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134416512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697414.803602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697414.803603:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697414.803605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134416512:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697415.379013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697416.403030:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697417.427019:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697418.451012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697419.475482:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697419.922982:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697419.922989:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697419.922992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697419.923001:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5321 00000100:00080000:3.0:1787697419.923011:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697419.923017:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697419.923048:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134416640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697419.923067:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697419.923070:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697419.923071:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697419.923074:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5321 00000100:00080000:3.0:1787697419.923076:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697419.923079:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697419.923089:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697419.923093:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134416640:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697419.923099:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134416768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697419.923107:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697419.923109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697419.923110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697419.923113:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5321 00000100:00080000:3.0:1787697419.923115:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697419.923117:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697419.923127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697419.923142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134416896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697419.923146:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697419.923157:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697419.923160:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697419.923162:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697419.923167:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5321 00000100:00080000:3.0:1787697419.923172:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697419.923175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697419.923181:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697419.923184:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697419/real 1787697419] req@ffff9d85c0fb3b80 x1874531134416640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697419.923197:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134417024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697419.923199:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134416640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697419.923224:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697419.923224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697419.923230:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697419.923232:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697419.923233:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697419.923241:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134416640:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697419.923241:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5321 00000100:00080000:3.0:1787697419.923243:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697419.923249:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697419.923254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697419.923266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134417152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697419.923271:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416768:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697419.923282:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697419.923287:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697419.923289:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697419.923295:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5321 00000800:00000100:1.0:1787697419.923296:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697419.923303:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697419.923309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697419/real 1787697419] req@ffff9d85c0fb0380 x1874531134416768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697419.923311:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697419.923320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134416768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697419.923329:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697419.923331:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697419.923332:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697419.923337:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697419.923342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697419.923349:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134417280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697419.923359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697419.923362:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697419.923366:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697419.923368:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697419.923373:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697419/real 1787697419] req@ffff9d85c0fb2d80 x1874531134416896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697419.923373:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5321 00000100:00080000:3.0:1787697419.923380:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697419.923387:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697419.923395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134416896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697419.923398:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697419.923409:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697419.923413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697419.923417:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134417408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697419.923420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134416896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697419.923432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697419.923454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697419.923464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697419/real 1787697419] req@ffff9d85c0fb1c00 x1874531134417024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697419.923479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134417024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697419.923486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697419.923489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697419.923492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697419.923507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134417152:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697419.923526:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697419.923534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697419/real 1787697419] req@ffff9d85c0fb3800 x1874531134417152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697419.923542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134417152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697419.923546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697419.923548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697419.923549:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134417152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697419.923553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134417280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697419.923562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697419.923566:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697419/real 1787697419] req@ffff9d85c0fb0e00 x1874531134417280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697419.923573:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134417280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697419.923577:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697419.923578:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697419.923580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134417280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697419.923583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134417408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697419.923590:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697419.923595:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697419/real 1787697419] req@ffff9d85c0fb0a80 x1874531134417408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697419.923601:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134417408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697419.923617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697419.923619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697419.923620:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134417408:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697420.499002:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697421.523061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697422.546991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697423.571006:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697424.594994:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697425.043012:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697425.043021:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697425.043025:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697425.043034:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5326 00000100:00080000:3.0:1787697425.043046:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697425.043052:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697425.043086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134417536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697425.043122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697425.043125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697425.043127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697425.043131:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5326 00000100:00080000:3.0:1787697425.043134:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697425.043138:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697425.043148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697425.043162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134417664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697425.043172:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134417536:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697425.043175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697425.043181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697425.043183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697425.043187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5326 00000100:00080000:3.0:1787697425.043191:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697425.043194:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697425.043200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697425.043212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134417792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697425.043220:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697425.043224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697425.043226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697425.043229:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5326 00000100:00080000:3.0:1787697425.043233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697425.043238:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697425.043241:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697425.043248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697425.043260:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134417920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697425.043269:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697425.043272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697425.043274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697425.043277:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5326 00000100:00080000:3.0:1787697425.043281:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697425.043286:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697425.043295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697425/real 1787697425] req@ffff9d85c0fb2a00 x1874531134417536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697425.043307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697425.043315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134417536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697425.043323:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134418048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697425.043330:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697425.043337:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697425.043337:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697425.043345:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697425.043346:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134417536:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697425.043349:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697425.043356:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5326 00000100:00080000:3.0:1787697425.043365:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697425.043370:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697425.043375:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697425.043380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697425.043389:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134418176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697425.043400:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697425.043404:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787697425.043407:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697425.043408:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697425.043416:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5326 00000100:00080000:3.0:1787697425.043420:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697425.043425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697425/real 1787697425] req@ffff9d85c0fb2680 x1874531134417664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697425.043426:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697425.043436:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697425.043444:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134417664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697425.043452:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134418304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697425.043457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697425.043465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697425.043472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697425.043479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697425.043500:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697425.043511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697425/real 1787697425] req@ffff9d85c0fb3100 x1874531134417792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697425.043522:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134417792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697425.043530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697425.043534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697425.043537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697425.043542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697425.043560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697425.043570:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697425/real 1787697425] req@ffff9d85c0fb3480 x1874531134417920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697425.043582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134417920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697425.043589:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697425.043592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697425.043595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134417920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697425.043601:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418048:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697425.043618:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697425.043629:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697425/real 1787697425] req@ffff9d85c0fb2300 x1874531134418048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697425.043640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134418048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697425.043648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697425.043651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697425.043654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418048:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697425.043679:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697425.043697:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697425.043708:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697425/real 1787697425] req@ffff9d85c0fb1880 x1874531134418176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697425.043719:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134418176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697425.043727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697425.043730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697425.043733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697425.043738:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697425.043755:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697425.043765:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697425/real 1787697425] req@ffff9d85c0fb1500 x1874531134418304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697425.043776:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134418304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697425.043801:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697425.043804:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697425.043808:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418304:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697425.619006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697426.643009:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697427.667000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697428.690987:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697429.714997:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697430.162922:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697430.162929:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697430.162932:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697430.162940:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5331 00000100:00080000:3.0:1787697430.162950:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697430.162956:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697430.162986:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134418432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697430.163009:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697430.163013:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697430.163014:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697430.163018:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5331 00000100:00080000:3.0:1787697430.163021:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697430.163025:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697430.163034:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697430.163047:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134418560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697430.163056:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697430.163058:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697430.163060:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697430.163064:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5331 00000100:00080000:3.0:1787697430.163067:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697430.163070:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697430.163075:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697430.163085:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134418688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697430.163094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697430.163098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697430.163100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697430.163104:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5331 00000100:00080000:3.0:1787697430.163107:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697430.163111:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697430.163116:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697430.163121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134418432:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697430.163139:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134418816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697430.163150:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697430.163154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697430.163156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697430.163159:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5331 00000100:00080000:3.0:1787697430.163164:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697430.163168:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697430.163174:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697430.163179:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697430.163189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134418944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697430.163198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697430.163200:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697430.163202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697430.163205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5331 00000100:00080000:3.0:1787697430.163208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697430.163210:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697430.163214:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697430.163226:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697430/real 1787697430] req@ffff9d85c0fb1f80 x1874531134418432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697430.163227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134419072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697430.163256:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697430.163259:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697430.163260:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697430.163263:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5331 00000100:00100000:1.0:1787697430.163264:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134418432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697430.163265:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697430.163271:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697430.163276:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697430.163279:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697430.163283:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697430.163286:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134419200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697430.163289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134418432:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697430.163305:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134418560:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697430.163322:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697430.163330:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697430/real 1787697430] req@ffff9d85c0fb0000 x1874531134418560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697430.163337:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134418560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697430.163342:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697430.163343:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697430.163345:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134418560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697430.163349:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134418688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697430.163359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697430.163364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697430/real 1787697430] req@ffff9d85c0fb0700 x1874531134418688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697430.163370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134418688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697430.163374:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697430.163376:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697430.163377:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134418688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697430.163394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134418816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697430.163403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697430.163409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697430/real 1787697430] req@ffff9d85c0fb1500 x1874531134418816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697430.163415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134418816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697430.163419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697430.163421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697430.163428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134418816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697430.163432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418944:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697430.163441:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697430.163447:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697430/real 1787697430] req@ffff9d85c0fb1880 x1874531134418944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697430.163454:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134418944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697430.163458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697430.163459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697430.163461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134418944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697430.163464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697430.163473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697430.163477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697430/real 1787697430] req@ffff9d85c0fb2300 x1874531134419072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697430.163484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134419072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697430.163488:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697430.163489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697430.163491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697430.163493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697430.163501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697430.163505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697430/real 1787697430] req@ffff9d85c0fb3480 x1874531134419200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697430.163510:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134419200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697430.163518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697430.163520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697430.163521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419200:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697430.738997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697431.762979:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697432.787004:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697433.811008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697434.835030:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697435.282963:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697435.282972:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697435.282976:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697435.282986:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5337 00000100:00080000:2.0:1787697435.282995:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697435.283002:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697435.283036:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134419328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697435.283062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697435.283066:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697435.283068:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697435.283071:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5337 00000100:00080000:2.0:1787697435.283074:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697435.283078:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697435.283086:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697435.283098:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134419456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697435.283104:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134419328:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697435.283114:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697435.283118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697435.283120:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697435.283125:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5337 00000100:00080000:2.0:1787697435.283130:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697435.283133:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697435.283138:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697435.283150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134419584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697435.283159:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697435.283162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697435.283164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697435.283167:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5337 00000800:00000100:1.0:1787697435.283173:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697435.283173:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697435.283181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697435.283185:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697435.283196:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134419712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697435.283205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697435.283208:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697435.283210:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697435.283213:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697435/real 1787697435] req@ffff9d85babd7480 x1874531134419328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697435.283216:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5337 00000100:00080000:2.0:1787697435.283223:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697435.283230:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697435.283235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697435.283240:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134419328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697435.283245:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134419840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697435.283261:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697435.283264:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697435.283268:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697435.283270:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697435.283272:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5337 00000100:00080000:1.0:1787697435.283276:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697435.283279:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697435.283280:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134419328:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697435.283287:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697435.283292:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697435.283302:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134419456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787697435.283305:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134419968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697435.283317:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697435.283321:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697435.283323:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697435.283324:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697435.283332:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5337 00000100:00080000:2.0:1787697435.283336:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697435.283340:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697435.283345:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697435.283349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697435/real 1787697435] req@ffff9d85babd5c00 x1874531134419456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697435.283360:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134420096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697435.283367:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134419456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697435.283380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697435.283383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697435.283385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134419456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697435.283391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134419584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697435.283419:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697435.283431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697435/real 1787697435] req@ffff9d85babd4000 x1874531134419584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697435.283442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134419584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697435.283450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697435.283453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697435.283456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134419584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697435.283462:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134419712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697435.283477:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697435.283486:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697435/real 1787697435] req@ffff9d85babd4380 x1874531134419712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697435.283497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134419712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697435.283505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697435.283508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697435.283511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134419712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697435.283517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419840:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697435.283532:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697435.283539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697435/real 1787697435] req@ffff9d85babd6300 x1874531134419840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697435.283546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134419840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697435.283551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697435.283553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697435.283554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697435.283565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697435.283574:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697435.283581:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697435/real 1787697435] req@ffff9d85babd6a00 x1874531134419968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697435.283591:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134419968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697435.283599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697435.283601:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697435.283604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134419968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697435.283608:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697435.283624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697435.283634:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697435/real 1787697435] req@ffff9d85babd6680 x1874531134420096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697435.283643:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134420096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697435.283654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697435.283657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697435.283659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420096:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697435.859034:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697436.883021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697437.907020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697438.931012:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697439.955129:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697440.403025:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697440.403062:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697440.403067:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697440.403080:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5342 00000100:00080000:3.0:1787697440.403094:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697440.403104:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697440.403152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134420224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697440.403188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697440.403193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697440.403195:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697440.403200:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5342 00000100:00080000:3.0:1787697440.403204:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697440.403209:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697440.403219:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697440.403232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134420352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697440.403254:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697440.403257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697440.403258:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697440.403262:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5342 00000100:00080000:3.0:1787697440.403264:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697440.403269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697440.403271:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134420224:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697440.403274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697440.403286:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134420480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697440.403293:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697440.403295:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697440.403296:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697440.403299:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5342 00000100:00080000:3.0:1787697440.403300:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697440.403302:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697440.403306:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697440.403313:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134420608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697440.403319:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697440.403322:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697440.403323:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697440.403325:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5342 00000100:00080000:3.0:1787697440.403327:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697440.403330:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697440.403332:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697440.403338:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134420736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697440.403344:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697440.403346:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697440.403347:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697440.403350:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5342 00000100:00080000:3.0:1787697440.403351:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697440.403356:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697440.403358:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697440.403360:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697440.403368:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134420864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697440.403374:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697440.403377:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697440.403379:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697440.403382:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5342 00000100:00080000:3.0:1787697440.403384:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697440.403386:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697440.403389:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697440.403395:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134420992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697440.403409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697440/real 1787697440] req@ffff9d85c0fb3100 x1874531134420224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697440.403425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134420224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697440.403442:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697440.403445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697440.403448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134420224:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697440.403454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134420352:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697440.403471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697440.403478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697440/real 1787697440] req@ffff9d85c0fb2680 x1874531134420352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697440.403486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134420352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697440.403491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697440.403493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697440.403495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134420352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697440.403515:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134420480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697440.403529:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697440.403538:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697440/real 1787697440] req@ffff9d85c0fb2a00 x1874531134420480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697440.403549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134420480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697440.403557:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697440.403565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697440.403569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134420480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697440.403575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134420608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697440.403626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697440.403639:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697440/real 1787697440] req@ffff9d85c0fb0a80 x1874531134420608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697440.403650:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134420608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697440.403659:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697440.403662:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697440.403665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134420608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697440.403671:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420736:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697440.403692:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697440.403703:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697440/real 1787697440] req@ffff9d85c0fb0e00 x1874531134420736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697440.403720:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134420736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697440.403727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697440.403730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697440.403733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697440.403739:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697440.403770:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697440.403783:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697440/real 1787697440] req@ffff9d85c0fb3800 x1874531134420864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697440.403794:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134420864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697440.403802:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697440.403805:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697440.403808:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697440.403814:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697440.403877:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697440.403891:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697440/real 1787697440] req@ffff9d85c0fb1c00 x1874531134420992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697440.403902:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134420992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697440.403916:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697440.403919:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697440.403922:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134420992:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697440.979066:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697442.003023:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697443.027034:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697444.051034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697445.075010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697445.522952:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697445.522962:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697445.522966:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697445.522975:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5347 00000100:00080000:3.0:1787697445.522985:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697445.522991:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697445.523025:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134421120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697445.523060:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697445.523063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697445.523066:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697445.523070:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5347 00000100:00080000:3.0:1787697445.523074:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697445.523078:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697445.523086:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697445.523098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134421248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697445.523106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697445.523108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697445.523110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697445.523113:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5347 00000100:00080000:3.0:1787697445.523116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697445.523119:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697445.523123:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697445.523131:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134421376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697445.523139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697445.523141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697445.523143:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697445.523145:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5347 00000100:00080000:3.0:1787697445.523148:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697445.523150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697445.523153:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697445.523165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134421504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697445.523172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697445.523174:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697445.523176:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697445.523179:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5347 00000100:00080000:3.0:1787697445.523182:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697445.523185:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697445.523189:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697445.523198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134421632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697445.523205:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697445.523208:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697445.523210:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697445.523213:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5347 00000100:00080000:3.0:1787697445.523215:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697445.523219:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697445.523223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697445.523232:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134421760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697445.523240:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697445.523243:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697445.523244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697445.523248:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5347 00000100:00080000:3.0:1787697445.523251:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697445.523254:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697445.523258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697445.523266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134421888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697445.523357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134421120:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697445.523449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134421248:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697445.523467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134421376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697445.523485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134421504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697445.523532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134421632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697445.523546:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134421760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697445.523555:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134421888:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697446.099037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697447.123026:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697448.147013:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697448.595134:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697448.595184:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697448.595201:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697448.595256:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697448.595268:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697448.595276:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697448.595285:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697448.595293:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697448.595312:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787697448.595329:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697445/real 1787697448] req@ffff9d85c0fb2d80 x1874531134421120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697448.595356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134421120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697448.595368:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697448.595373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697448.595378:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134421120:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697448.595388:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697445/real 1787697448] req@ffff9d85c0fb0380 x1874531134421248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697448.595401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134421248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697448.595408:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697448.595421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697448.595425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134421248:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697448.595432:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697445/real 1787697448] req@ffff9d85c0fb3b80 x1874531134421376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697448.595453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134421376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697448.595461:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697448.595464:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697448.595467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134421376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697448.595473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697445/real 1787697448] req@ffff9d85c0fb1c00 x1874531134421504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697448.595485:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134421504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697448.595493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697448.595496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697448.595504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134421504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697448.595510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697445/real 1787697448] req@ffff9d85c0fb3800 x1874531134421632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697448.595521:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134421632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697448.595528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697448.595531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697448.595534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134421632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697448.595540:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697445/real 1787697448] req@ffff9d85c0fb0e00 x1874531134421760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697448.595563:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134421760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697448.595571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697448.595574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697448.595577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134421760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697448.595584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697445/real 1787697448] req@ffff9d85c0fb0a80 x1874531134421888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697448.595593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134421888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697448.595600:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697448.595603:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697448.595606:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134421888:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697449.170962:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697450.195036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697450.642963:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697450.642972:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697450.642976:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697450.642987:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5352 00000100:00080000:3.0:1787697450.642997:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697450.643003:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697450.643034:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134422016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697450.643065:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697450.643069:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697450.643071:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697450.643075:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5352 00000100:00080000:3.0:1787697450.643078:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697450.643082:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697450.643090:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697450.643101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134422144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697450.643110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697450.643113:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697450.643114:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697450.643118:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5352 00000100:00080000:3.0:1787697450.643121:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697450.643125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697450.643130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697450.643139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134422272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697450.643147:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697450.643150:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697450.643151:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697450.643155:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5352 00000100:00080000:3.0:1787697450.643158:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697450.643161:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697450.643166:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697450.643175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134422400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697450.643184:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697450.643186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697450.643188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697450.643191:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5352 00000100:00080000:3.0:1787697450.643194:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697450.643197:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697450.643201:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697450.643208:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134422528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697450.643221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697450.643224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697450.643226:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697450.643229:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5352 00000100:00080000:3.0:1787697450.643232:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697450.643236:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697450.643240:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697450.643247:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134422656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697450.643255:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697450.643258:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697450.643259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697450.643262:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5352 00000100:00080000:3.0:1787697450.643264:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697450.643267:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697450.643271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697450.643278:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134422784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697450.643363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134422016:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787697450.643433:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697450.643470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697450/real 1787697450] req@ffff9d85c0fb2a00 x1874531134422016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697450.643485:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134422016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697450.643495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697450.643499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697450.643503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134422016:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697450.643509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134422144:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697450.643530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697450.643541:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697450/real 1787697450] req@ffff9d85c0fb2680 x1874531134422144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697450.643551:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134422144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697450.643559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697450.643562:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697450.643565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134422144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697450.643570:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134422272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697450.643585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697450.643594:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697450/real 1787697450] req@ffff9d85c0fb3100 x1874531134422272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697450.643609:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134422272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697450.643616:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697450.643619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697450.643622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134422272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697450.643627:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134422400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697450.643643:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697450.643651:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697450/real 1787697450] req@ffff9d85c0fb3480 x1874531134422400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697450.643667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134422400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697450.643674:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697450.643677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697450.643680:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134422400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697450.643685:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134422528:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697450.643703:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697450.643713:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697450/real 1787697450] req@ffff9d85c0fb2300 x1874531134422528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697450.643723:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134422528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697450.643735:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697450.643738:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697450.643740:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134422528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697450.643745:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134422656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697450.643763:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697450.643773:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697450/real 1787697450] req@ffff9d85c0fb1880 x1874531134422656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697450.643783:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134422656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697450.643791:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697450.643794:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697450.643797:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134422656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697450.643803:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134422784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697450.643821:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697450.643831:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697450/real 1787697450] req@ffff9d85c0fb1500 x1874531134422784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697450.643841:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134422784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697450.643848:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697450.643880:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697450.643884:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134422784:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697451.219042:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697452.243041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697453.267039:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697454.291015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697455.315031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697455.762964:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697455.762973:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697455.762978:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697455.762989:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5357 00000100:00080000:3.0:1787697455.763000:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697455.763007:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697455.763040:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134422912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697455.763072:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697455.763075:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697455.763077:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697455.763080:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5357 00000100:00080000:3.0:1787697455.763083:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697455.763087:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697455.763096:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697455.763106:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134423040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697455.763114:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697455.763117:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697455.763118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697455.763121:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5357 00000100:00080000:3.0:1787697455.763124:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697455.763127:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697455.763131:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697455.763138:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134423168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697455.763158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697455.763161:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697455.763163:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697455.763166:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5357 00000100:00080000:3.0:1787697455.763168:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697455.763171:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697455.763176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697455.763187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134423296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697455.763195:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697455.763197:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697455.763199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697455.763202:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5357 00000100:00080000:3.0:1787697455.763204:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697455.763208:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697455.763212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697455.763220:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134423424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697455.763227:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697455.763230:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697455.763231:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697455.763234:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5357 00000100:00080000:3.0:1787697455.763236:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697455.763240:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697455.763244:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697455.763251:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134423552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697455.763258:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697455.763261:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697455.763262:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697455.763265:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5357 00000100:00080000:3.0:1787697455.763267:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697455.763269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697455.763273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697455.763279:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134423680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697455.763380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134422912:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787697455.763443:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697455.763482:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697455/real 1787697455] req@ffff9d85c0fb0700 x1874531134422912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697455.763498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134422912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697455.763507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697455.763511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697455.763514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134422912:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697455.763521:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423040:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697455.763540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697455.763552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697455/real 1787697455] req@ffff9d85c0fb0000 x1874531134423040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697455.763562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134423040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697455.763570:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697455.763572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697455.763575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423040:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697455.763581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697455.763598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697455.763609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697455/real 1787697455] req@ffff9d85c0fb1f80 x1874531134423168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697455.763619:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134423168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697455.763626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697455.763628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697455.763631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697455.763636:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697455.763653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697455.763663:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697455/real 1787697455] req@ffff9d85c0fb1500 x1874531134423296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697455.763673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134423296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697455.763680:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697455.763684:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697455.763687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697455.763692:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134423424:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697455.763709:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697455.763718:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697455/real 1787697455] req@ffff9d85c0fb1880 x1874531134423424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697455.763728:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134423424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697455.763736:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697455.763738:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697455.763741:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134423424:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697455.763746:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134423552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697455.763762:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697455.763771:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697455/real 1787697455] req@ffff9d85c0fb2300 x1874531134423552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697455.763781:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134423552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697455.763788:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697455.763790:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697455.763793:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134423552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697455.763798:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134423680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697455.763814:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697455.763824:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697455/real 1787697455] req@ffff9d85c0fb3480 x1874531134423680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697455.763834:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134423680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697455.763841:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697455.763843:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697455.763846:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134423680:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697456.339027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697457.363017:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697458.387037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697459.411019:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697460.435028:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697460.882979:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697460.882988:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697460.882992:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697460.883002:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5362 00000100:00080000:3.0:1787697460.883012:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697460.883019:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697460.883060:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134423808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697460.883094:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697460.883099:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697460.883100:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697460.883104:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5362 00000100:00080000:3.0:1787697460.883109:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697460.883113:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697460.883122:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697460.883135:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134423936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697460.883147:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697460.883148:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134423808:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697460.883153:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697460.883157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697460.883162:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5362 00000100:00080000:3.0:1787697460.883165:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697460.883168:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697460.883173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697460.883184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134424064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697460.883193:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697460.883196:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697460.883197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697460.883201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5362 00000100:00080000:3.0:1787697460.883204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697460.883207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697460.883212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697460.883213:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697460.883224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134424192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697460.883261:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697460.883264:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697460.883266:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697460.883271:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5362 00000100:00100000:0.0:1787697460.883272:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697460/real 1787697460] req@ffff9d85c0fb3100 x1874531134423808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697460.883275:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697460.883283:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697460.883289:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697460.883299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134423808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697460.883301:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134424320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697460.883326:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697460.883326:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697460.883333:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697460.883336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697460.883337:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697460.883345:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5362 00000100:00080000:3.0:1787697460.883348:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697460.883351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134423808:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697460.883351:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697460.883354:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697460.883364:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134424448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697460.883370:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697460.883372:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697460.883374:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697460.883377:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5362 00000100:00080000:3.0:1787697460.883379:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697460.883384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423936:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697460.883384:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697460.883387:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697460.883396:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134424576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697460.883414:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697460.883426:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697460/real 1787697460] req@ffff9d85c0fb2680 x1874531134423936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697460.883435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134423936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697460.883440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697460.883442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697460.883445:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134423936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697460.883448:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697460.883460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697460.883466:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697460/real 1787697460] req@ffff9d85c0fb2a00 x1874531134424064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697460.883473:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134424064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697460.883477:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697460.883479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697460.883480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697460.883483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697460.883492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697460.883497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697460/real 1787697460] req@ffff9d85c0fb0a80 x1874531134424192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697460.883503:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134424192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697460.883508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697460.883509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697460.883511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697460.883513:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134424320:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697460.883521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697460.883526:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697460/real 1787697460] req@ffff9d85c0fb0e00 x1874531134424320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697460.883532:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134424320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697460.883537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697460.883539:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697460.883541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134424320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697460.883543:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134424448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697460.883551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697460.883556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697460/real 1787697460] req@ffff9d85c0fb3800 x1874531134424448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697460.883561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134424448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697460.883565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697460.883567:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697460.883569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134424448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697460.883579:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134424576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697460.883587:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697460.883592:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697460/real 1787697460] req@ffff9d85c0fb1c00 x1874531134424576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697460.883599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134424576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697460.883604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697460.883607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697460.883610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134424576:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697461.459015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697462.483053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697463.507037:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697464.531054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697465.555046:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697466.002971:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697466.002986:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697466.002991:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697466.003004:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5367 00000100:00080000:2.0:1787697466.003017:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697466.003026:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697466.003073:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134424704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697466.003103:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697466.003107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697466.003109:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697466.003114:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5367 00000100:00080000:2.0:1787697466.003118:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697466.003122:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697466.003134:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697466.003164:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134424832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697466.003297:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697466.003301:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697466.003303:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697466.003308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134424704:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697466.003309:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5367 00000100:00080000:2.0:1787697466.003317:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697466.003323:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697466.003328:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697466.003342:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134424960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697466.003353:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697466.003356:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697466.003358:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697466.003363:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5367 00000100:00080000:2.0:1787697466.003367:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697466.003371:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697466.003376:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697466.003391:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697466.003391:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134425088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697466.003408:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697466.003411:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697466.003413:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697466.003415:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5367 00000100:00080000:2.0:1787697466.003418:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697466.003423:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697466.003429:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697466.003441:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134425216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697466.003450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697466/real 1787697466] req@ffff9d85babd6d80 x1874531134424704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697466.003453:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697466.003462:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697466.003466:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697466.003472:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5367 00000100:00100000:0.0:1787697466.003478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134424704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697466.003484:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697466.003493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697466.003493:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697466.003501:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697466.003502:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697466.003509:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134424704:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697466.003520:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134425344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697466.003531:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697466.003536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424832:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697466.003536:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697466.003541:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697466.003546:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5367 00000100:00080000:2.0:1787697466.003549:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697466.003552:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697466.003557:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697466.003567:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134425472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697466.003574:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697466.003592:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697466/real 1787697466] req@ffff9d85babd5500 x1874531134424832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697466.003606:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134424832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697466.003624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697466.003628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697466.003631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697466.003637:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697466.003654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697466.003663:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697466/real 1787697466] req@ffff9d85babd7800 x1874531134424960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697466.003674:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134424960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697466.003682:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697466.003685:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697466.003688:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134424960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697466.003693:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697466.003705:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697466.003713:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697466/real 1787697466] req@ffff9d85babd6680 x1874531134425088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697466.003732:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134425088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697466.003739:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697466.003743:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697466.003745:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697466.003750:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134425216:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697466.003764:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697466.003772:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697466/real 1787697466] req@ffff9d85babd6a00 x1874531134425216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697466.003782:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134425216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697466.003790:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697466.003793:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697466.003796:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134425216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697466.003801:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134425344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697466.003814:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697466.003826:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697466/real 1787697466] req@ffff9d85babd6300 x1874531134425344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697466.003839:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134425344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697466.003846:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697466.003889:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697466.003892:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134425344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697466.003909:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134425472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697466.003926:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697466.003936:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697466/real 1787697466] req@ffff9d85babd4380 x1874531134425472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697466.003946:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134425472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697466.003964:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697466.003967:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697466.003970:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134425472:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697466.579020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697467.603027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697468.627025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697469.651006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697470.675022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697471.122968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697471.122977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697471.122981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697471.122991:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5372 00000100:00080000:3.0:1787697471.123001:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697471.123008:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697471.123040:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134425600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697471.123095:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697471.123099:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697471.123101:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697471.123105:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5372 00000100:00080000:3.0:1787697471.123108:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697471.123112:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697471.123122:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697471.123132:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134425600:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697471.123137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134425728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697471.123148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697471.123151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697471.123153:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697471.123162:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5372 00000100:00080000:3.0:1787697471.123165:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697471.123169:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697471.123174:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697471.123184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134425856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697471.123190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697471.123192:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697471.123194:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697471.123196:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5372 00000100:00080000:3.0:1787697471.123199:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697471.123201:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697471.123207:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697471.123212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697471.123227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134425984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697471.123236:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697471/real 1787697471] req@ffff9d85c0fb3b80 x1874531134425600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697471.123237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697471.123246:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697471.123249:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697471.123255:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5372 00000100:00080000:3.0:1787697471.123260:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697471.123263:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134425600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697471.123267:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697471.123276:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697471.123282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697471.123289:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697471.123289:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134426112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697471.123299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134425600:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697471.123301:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697471.123306:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697471.123309:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697471.123311:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5372 00000100:00080000:3.0:1787697471.123315:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697471.123318:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697471.123321:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697471.123327:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134426240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697471.123332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697471.123335:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697471.123336:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697471.123340:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5372 00000100:00100000:0.0:1787697471.123341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425728:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697471.123342:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697471.123347:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697471.123351:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697471.123358:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134426368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697471.123374:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697471.123393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697471/real 1787697471] req@ffff9d85c0fb0380 x1874531134425728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697471.123404:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134425728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697471.123412:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697471.123415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697471.123418:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697471.123424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697471.123441:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697471.123451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697471/real 1787697471] req@ffff9d85c0fb2d80 x1874531134425856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697471.123461:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134425856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697471.123469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697471.123472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697471.123475:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697471.123481:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697471.123499:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697471.123506:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697471/real 1787697471] req@ffff9d85c0fb1c00 x1874531134425984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697471.123513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134425984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697471.123518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697471.123519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697471.123521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134425984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697471.123525:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134426112:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697471.123545:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697471.123555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697471/real 1787697471] req@ffff9d85c0fb3800 x1874531134426112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697471.123576:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134426112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697471.123583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697471.123586:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697471.123588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134426112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697471.123594:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134426240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697471.123614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697471.123625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697471/real 1787697471] req@ffff9d85c0fb0e00 x1874531134426240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697471.123636:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134426240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697471.123642:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697471.123644:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697471.123646:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134426240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697471.123659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134426368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697471.123672:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697471.123679:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697471/real 1787697471] req@ffff9d85c0fb0a80 x1874531134426368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697471.123685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134426368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697471.123690:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697471.123691:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697471.123693:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134426368:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697471.699017:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697472.722991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697473.747043:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697474.771063:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697475.795112:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697476.243028:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697476.243041:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697476.243047:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697476.243060:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5378 00000100:00080000:0.0:1787697476.243073:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697476.243082:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697476.243140:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134426496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697476.243227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134426496:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787697476.243295:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697476.243329:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697476.243333:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697476.243335:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697476.243339:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5378 00000100:00080000:0.0:1787697476.243343:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697476.243346:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697476.243355:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697476.243366:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134426624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697476.243427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697476/real 1787697476] req@ffff9d85b9d0a300 x1874531134426496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697476.243447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134426496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697476.243458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697476.243462:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697476.243466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134426496:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697476.243491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134426624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697476.243508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697476.243516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697476/real 1787697476] req@ffff9d85b9d09180 x1874531134426624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697476.243523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134426624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697476.243529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697476.243531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697476.243533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134426624:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787697476.243577:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697476.243585:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697476.243587:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697476.243591:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5378 00000100:00080000:0.0:1787697476.243595:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697476.243600:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697476.243606:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697476.243618:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134426752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697476.243633:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697476.243637:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697476.243639:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697476.243642:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5378 00000100:00080000:0.0:1787697476.243646:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697476.243650:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697476.243654:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697476.243655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134426752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697476.243664:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134426880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697476.243673:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697476.243677:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787697476.243677:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697476.243679:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697476.243682:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5378 00000100:00080000:0.0:1787697476.243685:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697476.243687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697476/real 1787697476] req@ffff9d85b9d09500 x1874531134426752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697476.243689:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697476.243693:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697476.243700:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134426752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697476.243703:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134427008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697476.243708:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697476.243711:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697476.243711:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787697476.243715:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697476.243715:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134426752:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787697476.243716:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697476.243719:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5378 00000100:00080000:0.0:1787697476.243722:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697476.243725:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134426880:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787697476.243726:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697476.243730:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697476.243738:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134427136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697476.243742:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697476.243747:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697476.243750:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697476.243751:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697476/real 1787697476] req@ffff9d85b9d08700 x1874531134426880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697476.243752:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697476.243756:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5378 00000100:00080000:0.0:1787697476.243759:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697476.243762:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697476.243762:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134426880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697476.243767:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697476.243770:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697476.243773:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697476.243776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134426880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697476.243777:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134427264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697476.243782:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427008:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697476.243798:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697476.243809:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697476/real 1787697476] req@ffff9d85b9d0b800 x1874531134427008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697476.243821:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134427008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697476.243828:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697476.243831:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697476.243834:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697476.243847:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697476.243893:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697476.243900:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697476/real 1787697476] req@ffff9d85b9d09c00 x1874531134427136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697476.243908:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134427136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697476.243913:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697476.243915:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697476.243918:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697476.243924:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697476.243940:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697476.243950:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697476/real 1787697476] req@ffff9d85b9d09f80 x1874531134427264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697476.243961:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134427264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697476.243986:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697476.243990:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697476.243992:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427264:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697476.819832:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697477.843081:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697478.867008:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697479.891055:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697480.914996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697481.362985:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697481.362993:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697481.362997:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697481.363005:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5383 00000100:00080000:3.0:1787697481.363013:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697481.363018:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697481.363048:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134427392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697481.363071:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697481.363074:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697481.363076:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697481.363080:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5383 00000100:00080000:3.0:1787697481.363084:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697481.363087:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697481.363096:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697481.363106:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134427520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697481.363113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697481.363116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697481.363119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697481.363122:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134427392:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697481.363123:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5383 00000100:00080000:3.0:1787697481.363130:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697481.363135:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697481.363139:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697481.363152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134427648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697481.363161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697481.363165:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697481.363167:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697481.363171:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5383 00000100:00080000:3.0:1787697481.363174:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697481.363177:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697481.363181:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697481.363189:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697481.363190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134427776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697481.363200:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697481.363203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697481.363206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697481.363217:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5383 00000100:00080000:3.0:1787697481.363221:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697481.363225:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697481.363230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697481.363231:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697481/real 1787697481] req@ffff9d85c0fb2a00 x1874531134427392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697481.363245:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134427904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697481.363254:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134427392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697481.363256:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697481.363263:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697481.363267:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697481.363276:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5383 00000100:00080000:1.0:1787697481.363277:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697481.363279:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697481.363284:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697481.363287:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697481.363293:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134427392:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697481.363294:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697481.363308:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134428032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697481.363319:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697481.363322:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134427520:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697481.363323:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697481.363327:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697481.363333:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5383 00000100:00080000:3.0:1787697481.363337:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697481.363340:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697481.363346:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697481.363350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697481.363361:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134428160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697481.363368:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697481/real 1787697481] req@ffff9d85c0fb2680 x1874531134427520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697481.363384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134427520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697481.363394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697481.363398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697481.363401:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134427520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697481.363408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134427648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697481.363427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697481.363438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697481/real 1787697481] req@ffff9d85c0fb3100 x1874531134427648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697481.363449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134427648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697481.363457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697481.363460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697481.363463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134427648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697481.363469:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134427776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697481.363492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697481.363503:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697481/real 1787697481] req@ffff9d85c0fb3480 x1874531134427776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697481.363514:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134427776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697481.363522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697481.363525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697481.363529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134427776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697481.363534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427904:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697481.363552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697481.363563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697481/real 1787697481] req@ffff9d85c0fb2300 x1874531134427904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697481.363574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134427904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697481.363581:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697481.363584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697481.363587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134427904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697481.363602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697481.363619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697481.363635:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697481/real 1787697481] req@ffff9d85c0fb1880 x1874531134428032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697481.363647:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134428032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697481.363654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697481.363657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697481.363660:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697481.363665:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697481.363683:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697481.363693:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697481/real 1787697481] req@ffff9d85c0fb1500 x1874531134428160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697481.363703:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134428160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697481.363715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697481.363718:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697481.363721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428160:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697481.939023:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697482.963034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697483.987031:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697485.011040:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697486.034987:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697486.482927:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697486.482937:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697486.482942:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697486.482952:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5388 00000100:00080000:3.0:1787697486.482962:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697486.482969:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697486.483004:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134428288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697486.483032:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697486.483036:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697486.483037:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697486.483041:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5388 00000100:00080000:3.0:1787697486.483045:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697486.483048:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697486.483057:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697486.483070:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134428416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697486.483076:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134428288:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697486.483082:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697486.483086:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697486.483088:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697486.483091:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5388 00000100:00080000:3.0:1787697486.483094:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697486.483097:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697486.483102:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697486.483113:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134428544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697486.483129:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697486.483132:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697486.483134:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697486.483139:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697486.483141:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5388 00000100:00080000:3.0:1787697486.483147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697486.483153:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697486.483158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697486.483172:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134428672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697486.483180:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697486.483183:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697486.483185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697486.483191:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5388 00000100:00100000:1.0:1787697486.483192:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697486/real 1787697486] req@ffff9d85c0fb1f80 x1874531134428288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697486.483195:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697486.483203:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697486.483211:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697486.483219:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134428288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697486.483226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134428800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697486.483231:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697486.483240:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697486.483240:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697486.483245:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697486.483246:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134428288:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697486.483249:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697486.483256:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5388 00000100:00080000:3.0:1787697486.483260:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697486.483264:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697486.483268:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697486.483273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134428416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697486.483280:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134428928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697486.483292:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697486.483295:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697486.483299:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697486.483300:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697486.483307:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5388 00000100:00080000:3.0:1787697486.483312:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697486.483316:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697486.483319:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697486/real 1787697486] req@ffff9d85c0fb0000 x1874531134428416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697486.483322:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697486.483337:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134429056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697486.483340:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134428416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697486.483358:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697486.483362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697486.483366:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134428416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697486.483373:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134428544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697486.483390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697486.483400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697486/real 1787697486] req@ffff9d85c0fb0700 x1874531134428544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697486.483410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134428544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697486.483417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697486.483420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697486.483423:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134428544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697486.483428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134428672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697486.483440:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697486.483448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697486/real 1787697486] req@ffff9d85c0fb1500 x1874531134428672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697486.483458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134428672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697486.483466:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697486.483468:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697486.483471:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134428672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697486.483476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428800:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697486.483488:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697486.483495:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697486/real 1787697486] req@ffff9d85c0fb1880 x1874531134428800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697486.483506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134428800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697486.483513:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697486.483515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697486.483518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697486.483531:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697486.483544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697486.483551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697486/real 1787697486] req@ffff9d85c0fb2300 x1874531134428928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697486.483561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134428928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697486.483566:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697486.483567:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697486.483569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134428928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697486.483572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697486.483582:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697486.483587:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697486/real 1787697486] req@ffff9d85c0fb3480 x1874531134429056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697486.483593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134429056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697486.483602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697486.483603:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697486.483605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429056:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697487.059004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697488.083001:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697489.107012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697490.131015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697491.155020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697491.602940:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697491.602946:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697491.602950:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697491.602957:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5393 00000100:00080000:3.0:1787697491.602965:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697491.602971:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697491.603000:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134429184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697491.603019:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697491.603022:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697491.603023:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697491.603026:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5393 00000100:00080000:3.0:1787697491.603028:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697491.603030:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697491.603037:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697491.603043:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134429312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697491.603050:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697491.603052:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697491.603053:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697491.603055:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5393 00000100:00080000:3.0:1787697491.603057:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697491.603059:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697491.603062:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697491.603067:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134429184:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697491.603069:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134429440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697491.603079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697491.603081:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697491.603082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697491.603084:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5393 00000100:00080000:3.0:1787697491.603085:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697491.603087:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697491.603090:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697491.603097:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134429568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697491.603102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697491.603104:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697491.603105:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697491.603107:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5393 00000100:00080000:3.0:1787697491.603109:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697491.603111:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697491.603114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697491.603119:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134429696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697491.603124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697491.603125:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697491.603126:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697491.603128:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5393 00000100:00080000:3.0:1787697491.603131:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697491.603133:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697491.603135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697491.603140:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697491.603143:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134429824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697491.603149:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697491.603151:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697491.603153:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697491.603156:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5393 00000100:00080000:3.0:1787697491.603158:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697491.603160:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697491.603163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697491.603168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134429952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697491.603181:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697491/real 1787697491] req@ffff9d85c0fb3100 x1874531134429184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697491.603200:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134429184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697491.603209:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697491.603211:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697491.603214:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134429184:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697491.603219:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134429312:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697491.603234:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697491.603241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697491/real 1787697491] req@ffff9d85c0fb2680 x1874531134429312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697491.603252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134429312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697491.603259:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697491.603261:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697491.603264:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134429312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697491.603287:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134429440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697491.603302:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697491.603309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697491/real 1787697491] req@ffff9d85c0fb2a00 x1874531134429440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697491.603316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134429440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697491.603321:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697491.603322:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697491.603325:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134429440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697491.603328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134429568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697491.603338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697491.603346:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697491/real 1787697491] req@ffff9d85c0fb0a80 x1874531134429568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697491.603355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134429568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697491.603362:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697491.603365:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697491.603368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134429568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697491.603374:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429696:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697491.603389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697491.603396:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697491/real 1787697491] req@ffff9d85c0fb0e00 x1874531134429696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697491.603403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134429696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697491.603407:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697491.603409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697491.603411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697491.603414:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697491.603422:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697491.603427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697491/real 1787697491] req@ffff9d85c0fb3800 x1874531134429824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697491.603434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134429824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697491.603438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697491.603439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697491.603441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697491.603444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697491.603452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697491.603456:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697491/real 1787697491] req@ffff9d85c0fb1c00 x1874531134429952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697491.603462:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134429952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697491.603470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697491.603472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697491.603473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134429952:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697492.179005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697493.203035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697494.227016:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697495.250993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697496.274997:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697496.722962:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697496.722972:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697496.722976:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697496.722987:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5398 00000100:00080000:3.0:1787697496.722997:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697496.723004:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697496.723039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134430080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697496.723068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697496.723072:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697496.723074:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697496.723079:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5398 00000100:00080000:3.0:1787697496.723082:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697496.723086:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697496.723095:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697496.723108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134430208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697496.723116:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134430080:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697496.723123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697496.723127:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697496.723129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697496.723134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5398 00000100:00080000:3.0:1787697496.723137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697496.723141:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697496.723147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697496.723159:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134430336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697496.723168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697496.723171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697496.723173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697496.723177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5398 00000100:00080000:3.0:1787697496.723181:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697496.723186:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787697496.723190:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697496.723191:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697496.723207:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134430464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697496.723216:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697496.723220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697496.723222:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697496.723225:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5398 00000100:00080000:3.0:1787697496.723229:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697496.723234:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697496.723239:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697496/real 1787697496] req@ffff9d85c0fb2d80 x1874531134430080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697496.723239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697496.723256:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134430592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697496.723263:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134430080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697496.723274:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697496.723280:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697496.723284:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697496.723285:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697496.723288:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697496.723292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134430080:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697496.723297:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5398 00000100:00080000:3.0:1787697496.723302:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697496.723306:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697496.723311:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697496.723318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134430208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697496.723325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134430720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697496.723336:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697496.723341:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697496.723341:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697496.723345:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697496.723351:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5398 00000100:00080000:3.0:1787697496.723355:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697496.723356:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697496/real 1787697496] req@ffff9d85c0fb0380 x1874531134430208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697496.723363:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697496.723373:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697496.723378:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134430208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697496.723389:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697496.723389:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134430848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697496.723396:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697496.723404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134430208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697496.723415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134430336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697496.723432:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697496.723442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697496/real 1787697496] req@ffff9d85c0fb3b80 x1874531134430336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697496.723452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134430336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697496.723460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697496.723463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697496.723466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134430336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697496.723470:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134430464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697496.723483:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697496.723490:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697496/real 1787697496] req@ffff9d85c0fb1c00 x1874531134430464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697496.723500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134430464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697496.723507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697496.723509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697496.723512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134430464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697496.723516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134430592:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697496.723529:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697496.723536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697496/real 1787697496] req@ffff9d85c0fb3800 x1874531134430592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697496.723546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134430592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697496.723553:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697496.723556:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697496.723559:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134430592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697496.723571:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134430720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697496.723583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697496.723591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697496/real 1787697496] req@ffff9d85c0fb0e00 x1874531134430720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697496.723601:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134430720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697496.723608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697496.723623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697496.723627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134430720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697496.723632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134430848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697496.723646:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697496.723655:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697496/real 1787697496] req@ffff9d85c0fb0a80 x1874531134430848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697496.723665:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134430848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697496.723678:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697496.723681:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697496.723684:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134430848:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697497.299002:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697498.322978:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697499.347026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697500.370974:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697501.395024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697501.842970:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697501.842979:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697501.842983:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697501.842992:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5403 00000100:00080000:2.0:1787697501.843002:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697501.843008:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697501.843050:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134430976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697501.843076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697501.843079:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697501.843081:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697501.843085:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5403 00000100:00080000:2.0:1787697501.843088:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697501.843092:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697501.843100:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697501.843112:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134431104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697501.843122:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134430976:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697501.843124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697501.843133:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697501.843135:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697501.843139:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5403 00000100:00080000:2.0:1787697501.843142:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697501.843146:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697501.843157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697501.843168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134431232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697501.843177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697501.843180:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697501.843182:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697501.843187:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697501.843191:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5403 00000100:00080000:2.0:1787697501.843195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697501.843199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697501.843203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697501.843213:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134431360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697501.843221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697501.843224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697501.843226:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697501.843229:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5403 00000100:00080000:2.0:1787697501.843233:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697501.843236:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697501.843240:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697501.843250:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697501/real 1787697501] req@ffff9d85babd4000 x1874531134430976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697501.843260:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134431488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697501.843271:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697501.843275:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134430976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697501.843276:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697501.843279:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697501.843287:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5403 00000100:00080000:2.0:1787697501.843293:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697501.843295:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697501.843298:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697501.843303:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697501.843306:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697501.843309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134430976:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697501.843319:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134431616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697501.843328:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697501.843331:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697501.843334:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134431104:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697501.843334:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697501.843343:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5403 00000100:00080000:2.0:1787697501.843348:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697501.843351:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697501.843356:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697501.843363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697501.843368:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134431744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697501.843378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697501/real 1787697501] req@ffff9d85babd5c00 x1874531134431104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697501.843391:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134431104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697501.843399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697501.843402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697501.843405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134431104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697501.843412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134431232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697501.843428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697501.843438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697501/real 1787697501] req@ffff9d85babd7480 x1874531134431232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697501.843449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134431232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697501.843456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697501.843459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697501.843462:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134431232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697501.843467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134431360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697501.843482:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697501.843490:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697501/real 1787697501] req@ffff9d85babd5180 x1874531134431360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697501.843500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134431360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697501.843506:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697501.843509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697501.843512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134431360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697501.843517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134431488:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697501.843562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697501.843571:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697501/real 1787697501] req@ffff9d85babd4a80 x1874531134431488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697501.843581:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134431488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697501.843589:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697501.843591:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697501.843594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134431488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697501.843599:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134431616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697501.843614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697501.843622:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697501/real 1787697501] req@ffff9d85babd7b80 x1874531134431616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697501.843631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134431616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697501.843637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697501.843639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697501.843642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134431616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697501.843657:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134431744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697501.843672:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697501.843680:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697501/real 1787697501] req@ffff9d85babd7100 x1874531134431744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697501.843691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134431744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697501.843704:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697501.843708:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697501.843711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134431744:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697502.419117:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697503.443024:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697504.467030:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697505.491013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697506.515051:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697506.963002:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697506.963012:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697506.963017:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697506.963029:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5408 00000100:00080000:0.0:1787697506.963044:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697506.963051:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697506.963089:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134431872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697506.963114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697506.963116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697506.963118:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697506.963120:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5408 00000100:00080000:0.0:1787697506.963122:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697506.963125:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697506.963135:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697506.963149:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134432000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697506.963160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697506.963163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697506.963166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697506.963169:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134431872:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697506.963172:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5408 00000100:00080000:0.0:1787697506.963180:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697506.963185:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697506.963190:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697506.963204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134432128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697506.963214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697506.963218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697506.963220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697506.963224:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5408 00000100:00080000:0.0:1787697506.963242:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697506.963249:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787697506.963250:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787697506.963256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697506.963270:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134432256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697506.963288:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697506.963292:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697506.963294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697506.963299:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5408 00000100:00080000:0.0:1787697506.963303:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697506.963305:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697506/real 1787697506] req@ffff9d85b9d0b480 x1874531134431872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697506.963310:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697506.963321:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697506.963332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134431872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697506.963338:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134432384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697506.963353:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697506.963355:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697506.963360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787697506.963362:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697506.963366:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697506.963369:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134431872:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787697506.963373:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5408 00000100:00080000:0.0:1787697506.963389:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697506.963394:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697506.963399:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432000:192.168.201.137@tcp:38:kworker.0 00000008:00100000:0.0:1787697506.963402:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697506.963418:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134432512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697506.963449:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697506.963452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787697506.963454:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697506.963458:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697506.963463:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5408 00000100:00080000:0.0:1787697506.963468:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697506.963470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697506/real 1787697506] req@ffff9d85b9d0a680 x1874531134432000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697506.963475:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787697506.963485:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697506.963491:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134432000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697506.963501:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134432640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697506.963502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697506.963509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697506.963516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697506.963524:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697506.963549:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697506.963562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697506/real 1787697506] req@ffff9d85b9d0bb80 x1874531134432128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697506.963572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134432128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697506.963579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697506.963582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697506.963585:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697506.963590:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697506.963605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697506.963614:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697506/real 1787697506] req@ffff9d85b9d0ad80 x1874531134432256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697506.963624:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134432256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697506.963632:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697506.963634:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697506.963637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697506.963642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134432384:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697506.963659:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697506.963668:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697506/real 1787697506] req@ffff9d85b9d08000 x1874531134432384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697506.963678:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134432384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697506.963685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697506.963688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697506.963690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134432384:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697506.963705:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134432512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697506.963720:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697506.963734:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697506/real 1787697506] req@ffff9d85b9d08a80 x1874531134432512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697506.963745:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134432512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697506.963752:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697506.963755:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697506.963758:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134432512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697506.963763:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134432640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697506.963778:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697506.963787:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697506/real 1787697506] req@ffff9d85b9d08e00 x1874531134432640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697506.963797:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134432640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697506.963810:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697506.963812:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697506.963815:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134432640:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697507.539008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697508.563036:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697509.587013:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697510.611010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697511.635052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697512.083014:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697512.083028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697512.083033:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697512.083044:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5413 00000100:00080000:3.0:1787697512.083066:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697512.083075:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697512.083123:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134432768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697512.083157:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697512.083161:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697512.083163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697512.083168:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5413 00000100:00080000:3.0:1787697512.083172:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697512.083176:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697512.083187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697512.083201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134432896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697512.083210:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697512.083214:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697512.083216:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697512.083219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5413 00000100:00080000:3.0:1787697512.083222:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697512.083226:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697512.083231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697512.083241:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134433024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697512.083250:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697512.083254:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697512.083256:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697512.083259:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5413 00000100:00080000:3.0:1787697512.083262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697512.083265:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697512.083269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697512.083279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134433152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697512.083285:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697512.083293:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697512.083294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697512.083296:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5413 00000100:00080000:3.0:1787697512.083299:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697512.083302:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697512.083307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697512.083317:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134433280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697512.083325:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697512.083328:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697512.083330:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697512.083334:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5413 00000100:00080000:3.0:1787697512.083337:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697512.083341:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697512.083345:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697512.083380:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134433408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697512.083386:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697512.083388:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697512.083390:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697512.083392:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5413 00000100:00080000:3.0:1787697512.083394:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697512.083396:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697512.083399:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697512.083405:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134433536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697512.086075:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134432768:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697512.086211:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697512.086239:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697512.086255:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697512.086279:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134433280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697512.086297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134433408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697512.086314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134433536:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697512.659078:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697513.683104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697514.707074:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697515.155256:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697515.155328:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697515.155349:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697515.155401:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697515.155411:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697515.155429:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697515.155437:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697515.155445:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697515.155453:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787697515.155564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697512/real 1787697515] req@ffff9d85c0fb2a00 x1874531134432768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697515.155593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134432768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697515.155605:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697515.155611:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697515.155616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134432768:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697515.155624:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697512/real 1787697515] req@ffff9d85c0fb2680 x1874531134432896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697515.155638:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134432896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697515.155647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697515.155651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697515.155655:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134432896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697515.155662:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697512/real 1787697515] req@ffff9d85c0fb3100 x1874531134433024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697515.155686:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134433024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697515.155698:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697515.155702:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697515.155705:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697515.155713:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697512/real 1787697515] req@ffff9d85c0fb3480 x1874531134433152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697515.155725:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134433152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697515.155733:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697515.155736:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697515.155740:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697515.155747:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697512/real 1787697515] req@ffff9d85c0fb2300 x1874531134433280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697515.155760:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134433280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697515.155767:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697515.155770:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697515.155773:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134433280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697515.155780:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697512/real 1787697515] req@ffff9d85c0fb1880 x1874531134433408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697515.155792:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134433408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697515.155800:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697515.155804:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697515.155807:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134433408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697515.155815:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697512/real 1787697515] req@ffff9d85c0fb1500 x1874531134433536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697515.155825:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134433536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697515.155834:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697515.155837:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697515.155840:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134433536:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697515.731079:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697516.755018:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697517.202962:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697517.202972:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697517.202976:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697517.202985:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5418 00000100:00080000:3.0:1787697517.203004:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697517.203011:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697517.203042:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134433664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697517.203069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697517.203073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697517.203074:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697517.203078:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5418 00000100:00080000:3.0:1787697517.203082:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697517.203086:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697517.203095:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697517.203106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134433664:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697517.203115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134433792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697517.203126:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697517.203130:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697517.203131:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697517.203135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5418 00000100:00080000:3.0:1787697517.203139:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697517.203143:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697517.203148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697517.203160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134433920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697517.203169:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697517.203173:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697517.203178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697517.203180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697517.203184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5418 00000100:00080000:3.0:1787697517.203189:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697517.203193:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697517.203198:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697517.203214:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697517/real 1787697517] req@ffff9d85c0fb0700 x1874531134433664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697517.203215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134434048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697517.203238:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697517.203241:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697517.203243:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697517.203249:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134433664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697517.203249:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5418 00000100:00080000:3.0:1787697517.203255:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697517.203275:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697517.203280:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697517.203290:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134434176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697517.203292:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697517.203302:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697517.203306:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697517.203309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134433664:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697517.203311:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697517.203315:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697517.203320:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5418 00000100:00080000:3.0:1787697517.203323:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697517.203327:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697517.203331:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697517.203339:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134434304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697517.203343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433792:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697517.203356:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697517.203359:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697517.203361:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697517.203365:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5418 00000100:00080000:3.0:1787697517.203369:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697517.203372:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787697517.203377:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697517.203379:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697517.203392:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134434432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697517.203394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697517/real 1787697517] req@ffff9d85c0fb0000 x1874531134433792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697517.203422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134433792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697517.203431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697517.203442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697517.203447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697517.203454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697517.203474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697517.203486:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697517/real 1787697517] req@ffff9d85c0fb1f80 x1874531134433920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697517.203517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134433920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697517.203526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697517.203528:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697517.203532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134433920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697517.203538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697517.203559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697517.203570:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697517/real 1787697517] req@ffff9d85c0fb1500 x1874531134434048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697517.203581:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134434048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697517.203589:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697517.203592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697517.203595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697517.203601:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134434176:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697517.203621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697517.203648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697517/real 1787697517] req@ffff9d85c0fb1880 x1874531134434176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697517.203659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134434176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697517.203672:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697517.203675:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697517.203678:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134434176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697517.203696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134434304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697517.203717:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697517.203729:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697517/real 1787697517] req@ffff9d85c0fb2300 x1874531134434304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697517.203740:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134434304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697517.203749:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697517.203752:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697517.203755:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134434304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697517.203761:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134434432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697517.203780:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697517.203791:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697517/real 1787697517] req@ffff9d85c0fb3480 x1874531134434432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697517.203803:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134434432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697517.203811:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697517.203814:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697517.203820:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134434432:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697517.779084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697518.802997:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697519.827001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697520.851024:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697521.874973:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697522.322969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697522.322977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697522.322980:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697522.322988:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5424 00000100:00080000:1.0:1787697522.322998:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697522.323004:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697522.323046:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134434560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697522.323067:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697522.323069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697522.323071:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697522.323073:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5424 00000100:00080000:1.0:1787697522.323076:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697522.323078:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697522.323085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697522.323093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134434688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697522.323100:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697522.323102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697522.323103:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697522.323106:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5424 00000100:00080000:1.0:1787697522.323107:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697522.323110:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697522.323114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697522.323116:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134434560:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697522.323122:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134434816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697522.323129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697522.323131:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697522.323132:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697522.323134:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5424 00000100:00080000:1.0:1787697522.323135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697522.323137:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697522.323140:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697522.323148:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134434944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697522.323157:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697522.323160:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697522.323162:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697522.323165:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5424 00000100:00080000:1.0:1787697522.323168:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697522.323183:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697522.323188:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697522.323191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697522.323205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134435072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697522.323212:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697522.323214:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697522.323216:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697522.323218:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5424 00000100:00080000:1.0:1787697522.323222:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697522.323226:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697522.323228:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697522/real 1787697522] req@ffff9d85b9d2f800 x1874531134434560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697522.323230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697522.323239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134435200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697522.323246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697522.323248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697522.323250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697522.323253:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5424 00000100:00080000:1.0:1787697522.323255:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697522.323257:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697522.323260:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697522.323262:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134434560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697522.323268:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134435328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697522.323277:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697522.323283:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697522.323289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134434560:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697522.323298:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434688:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697522.323323:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697522.323334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697522/real 1787697522] req@ffff9d85b9d2dc00 x1874531134434688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697522.323347:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134434688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697522.323355:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697522.323357:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697522.323361:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697522.323384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697522.323402:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697522.323413:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697522/real 1787697522] req@ffff9d85b9d2e680 x1874531134434816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697522.323425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134434816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697522.323438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697522.323441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697522.323444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697522.323450:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697522.323469:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697522.323479:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697522/real 1787697522] req@ffff9d85b9d2d500 x1874531134434944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697522.323490:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134434944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697522.323505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697522.323507:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697522.323511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134434944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697522.323523:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435072:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697522.323543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697522.323553:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697522/real 1787697522] req@ffff9d85b9d2ed80 x1874531134435072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697522.323564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134435072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697522.323569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697522.323571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697522.323573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697522.323577:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697522.323595:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697522.323602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697522/real 1787697522] req@ffff9d85b9d2c700 x1874531134435200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697522.323609:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134435200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697522.323616:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697522.323620:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697522.323622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697522.323625:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697522.323637:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697522.323643:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697522/real 1787697522] req@ffff9d85b9d2c380 x1874531134435328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697522.323652:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134435328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697522.323659:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697522.323661:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697522.323664:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435328:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697522.899003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697523.923018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697524.946993:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697525.971016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697526.995003:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697527.442975:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697527.442987:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697527.442992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697527.443005:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5429 00000100:00080000:3.0:1787697527.443017:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697527.443033:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697527.443078:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134435456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697527.443144:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697527.443148:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697527.443150:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697527.443154:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5429 00000100:00080000:3.0:1787697527.443158:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697527.443162:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697527.443173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697527.443186:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134435584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697527.443196:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697527.443198:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134435456:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697527.443201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697527.443205:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697527.443211:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5429 00000100:00080000:3.0:1787697527.443215:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697527.443218:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697527.443224:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697527.443235:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134435712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697527.443245:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697527.443248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697527.443250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697527.443254:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5429 00000100:00080000:3.0:1787697527.443258:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697527.443262:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697527.443267:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697527.443268:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697527.443280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134435840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697527.443290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697527.443294:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697527.443295:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697527.443298:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5429 00000100:00080000:3.0:1787697527.443301:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697527.443305:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697527.443309:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697527.443313:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697527/real 1787697527] req@ffff9d85c0fb3100 x1874531134435456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697527.443322:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134435968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697527.443334:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697527.443338:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697527.443340:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134435456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697527.443341:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697527.443348:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5429 00000100:00080000:3.0:1787697527.443353:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697527.443359:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697527.443363:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697527.443364:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697527.443373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697527.443377:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134436096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697527.443381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134435456:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697527.443399:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697527.443402:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697527.443404:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697527.443408:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5429 00000100:00080000:3.0:1787697527.443412:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697527.443417:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134435584:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697527.443418:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697527.443427:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697527.443442:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134436224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697527.443452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697527.443468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697527/real 1787697527] req@ffff9d85c0fb2680 x1874531134435584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697527.443481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134435584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697527.443490:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697527.443493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697527.443497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134435584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697527.443503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134435712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697527.443524:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697527.443532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697527/real 1787697527] req@ffff9d85c0fb2a00 x1874531134435712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697527.443542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134435712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697527.443550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697527.443553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697527.443556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134435712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697527.443562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134435840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697527.443581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697527.443591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697527/real 1787697527] req@ffff9d85c0fb0a80 x1874531134435840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697527.443602:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134435840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697527.443611:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697527.443614:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697527.443617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134435840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697527.443623:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435968:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697527.443643:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697527.443652:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697527/real 1787697527] req@ffff9d85c0fb0e00 x1874531134435968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697527.443659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134435968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697527.443664:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697527.443666:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697527.443668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134435968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697527.443671:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697527.443683:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697527.443688:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697527/real 1787697527] req@ffff9d85c0fb3800 x1874531134436096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697527.443694:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134436096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697527.443698:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697527.443700:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697527.443702:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697527.443714:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697527.443723:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697527.443728:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697527/real 1787697527] req@ffff9d85c0fb1c00 x1874531134436224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697527.443735:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134436224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697527.443739:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697527.443741:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697527.443743:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436224:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697528.019015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697529.042959:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697530.067009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697531.091075:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697532.114986:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697532.562924:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697532.562934:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697532.562937:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697532.562948:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5434 00000100:00080000:3.0:1787697532.562964:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697532.562971:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697532.563031:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134436352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697532.563058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697532.563063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697532.563065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697532.563068:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5434 00000100:00080000:3.0:1787697532.563071:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697532.563075:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697532.563085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697532.563095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134436480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697532.563102:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697532.563105:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697532.563107:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697532.563115:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134436352:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697532.563116:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5434 00000100:00080000:3.0:1787697532.563128:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697532.563134:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697532.563141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697532.563156:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134436608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697532.563166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697532.563169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697532.563171:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697532.563175:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5434 00000100:00080000:3.0:1787697532.563179:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697532.563184:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697532.563190:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697532.563194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697532.563215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134436736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697532.563225:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697532.563229:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697532.563231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697532.563234:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697532/real 1787697532] req@ffff9d85c0fb3b80 x1874531134436352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697532.563240:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5434 00000100:00080000:3.0:1787697532.563252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697532.563261:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697532.563268:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134436352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697532.563271:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697532.563288:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697532.563298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697532.563299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134436864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697532.563303:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134436352:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697532.563325:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697532.563328:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697532.563330:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697532.563333:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5434 00000100:00080000:3.0:1787697532.563336:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697532.563339:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697532.563344:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697532.563350:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134436480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697532.563357:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134436992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697532.563365:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697532.563368:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697532.563369:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697532.563372:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5434 00000100:00080000:3.0:1787697532.563375:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697532.563377:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697532.563382:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697532.563391:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697532.563392:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134437120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697532.563420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697532/real 1787697532] req@ffff9d85c0fb0380 x1874531134436480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697532.563434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134436480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697532.563440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697532.563443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697532.563445:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134436480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697532.563450:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134436608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697532.563467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697532.563473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697532/real 1787697532] req@ffff9d85c0fb2d80 x1874531134436608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697532.563480:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134436608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697532.563485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697532.563487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697532.563490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134436608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697532.563496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134436736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697532.563512:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697532.563522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697532/real 1787697532] req@ffff9d85c0fb1c00 x1874531134436736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697532.563532:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134436736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697532.563541:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697532.563544:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697532.563547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134436736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697532.563554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436864:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697532.563572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697532.563581:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697532/real 1787697532] req@ffff9d85c0fb3800 x1874531134436864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697532.563588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134436864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697532.563593:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697532.563596:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697532.563597:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697532.563610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697532.563622:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697532.563627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697532/real 1787697532] req@ffff9d85c0fb0e00 x1874531134436992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697532.563634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134436992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697532.563638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697532.563640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697532.563642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134436992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697532.563645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134437120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697532.563661:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697532.563666:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697532/real 1787697532] req@ffff9d85c0fb0a80 x1874531134437120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697532.563673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134437120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697532.563677:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697532.563679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697532.563681:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134437120:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697533.139066:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697534.163037:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697535.187024:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697536.211019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697537.235005:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697537.682955:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697537.682962:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697537.682966:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697537.682973:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5439 00000100:00080000:3.0:1787697537.682980:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697537.682986:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697537.683014:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134437248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697537.683036:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697537.683040:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697537.683041:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697537.683045:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5439 00000100:00080000:3.0:1787697537.683048:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697537.683052:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697537.683061:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697537.683071:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134437376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697537.683080:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697537.683083:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697537.683085:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697537.683086:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134437248:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697537.683089:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5439 00000100:00080000:3.0:1787697537.683092:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697537.683095:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697537.683099:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697537.683106:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134437504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697537.683113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697537.683116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697537.683117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697537.683120:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5439 00000100:00080000:3.0:1787697537.683122:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697537.683124:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697537.683128:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697537.683135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134437632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697537.683139:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697537.683142:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697537.683145:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697537.683146:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697537.683149:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5439 00000100:00080000:3.0:1787697537.683152:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697537.683154:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697537.683158:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697537.683166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134437760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697537.683174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697537.683176:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697537/real 1787697537] req@ffff9d85c0fb2a00 x1874531134437248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697537.683177:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697537.683178:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697537.683181:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5439 00000100:00080000:3.0:1787697537.683184:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697537.683187:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697537.683191:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134437248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697537.683191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697537.683198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134437888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697537.683201:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697537.683204:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697537.683205:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697537.683208:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134437248:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697537.683208:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697537.683210:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697537.683212:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5439 00000100:00080000:3.0:1787697537.683215:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697537.683217:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697537.683220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697537.683227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134437376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697537.683227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134438016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697537.683245:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697537.683257:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697537/real 1787697537] req@ffff9d85c0fb2680 x1874531134437376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697537.683268:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134437376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697537.683276:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697537.683278:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697537.683280:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134437376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697537.683284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134437504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697537.683297:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697537.683304:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697537/real 1787697537] req@ffff9d85c0fb3100 x1874531134437504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697537.683311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134437504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697537.683315:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697537.683316:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697537.683318:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134437504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697537.683321:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134437632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697537.683330:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697537.683334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697537/real 1787697537] req@ffff9d85c0fb3480 x1874531134437632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697537.683340:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134437632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697537.683345:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697537.683347:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697537.683349:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134437632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697537.683352:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134437760:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697537.683360:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697537.683364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697537/real 1787697537] req@ffff9d85c0fb2300 x1874531134437760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697537.683370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134437760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697537.683374:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697537.683375:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697537.683377:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134437760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697537.683380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134437888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697537.683392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697537.683401:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697537/real 1787697537] req@ffff9d85c0fb1880 x1874531134437888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697537.683410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134437888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697537.683417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697537.683420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697537.683424:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134437888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697537.683438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697537.683452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697537.683461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697537/real 1787697537] req@ffff9d85c0fb1500 x1874531134438016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697537.683471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134438016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697537.683478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697537.683481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697537.683484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438016:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697538.259030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697539.282985:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697540.306997:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697541.331022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697542.355025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697542.802962:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697542.802973:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697542.802977:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697542.802987:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5444 00000100:00080000:1.0:1787697542.802998:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697542.803005:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697542.803040:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134438144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697542.803069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697542.803073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697542.803075:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697542.803079:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5444 00000100:00080000:1.0:1787697542.803083:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697542.803087:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697542.803096:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697542.803110:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134438272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697542.803116:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697542.803119:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697542.803120:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697542.803123:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5444 00000100:00080000:1.0:1787697542.803126:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697542.803129:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697542.803134:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697542.803145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134438400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697542.803147:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134438144:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697542.803163:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697542.803167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697542.803168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697542.803173:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5444 00000100:00080000:1.0:1787697542.803177:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697542.803181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697542.803186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697542.803198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134438528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697542.803206:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697542.803209:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697542.803211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697542.803216:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697542.803219:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5444 00000100:00080000:1.0:1787697542.803224:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697542.803229:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697542.803234:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697542.803246:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134438656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697542.803255:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697542.803259:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697542.803260:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697542.803265:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5444 00000100:00100000:0.0:1787697542.803266:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697542/real 1787697542] req@ffff9d85b9d2fb80 x1874531134438144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697542.803268:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697542.803277:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697542.803286:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697542.803294:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134438144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697542.803299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134438784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697542.803311:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697542.803318:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697542.803325:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697542.803330:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134438144:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697542.803331:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697542.803337:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697542.803341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134438272:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697542.803345:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5444 00000100:00080000:1.0:1787697542.803350:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697542.803364:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697542.803368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697542.803372:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697542.803379:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697542/real 1787697542] req@ffff9d85b9d2f480 x1874531134438272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697542.803387:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134438912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697542.803390:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134438272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697542.803411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697542.803415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697542.803418:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134438272:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697542.803442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134438400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697542.803462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697542.803473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697542/real 1787697542] req@ffff9d85b9d2ca80 x1874531134438400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697542.803485:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134438400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697542.803492:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697542.803495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697542.803498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134438400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697542.803504:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134438528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697542.803522:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697542.803533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697542/real 1787697542] req@ffff9d85b9d2c000 x1874531134438528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697542.803544:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134438528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697542.803552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697542.803555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697542.803558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134438528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697542.803563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438656:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697542.803583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697542.803593:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697542/real 1787697542] req@ffff9d85b9d2ce00 x1874531134438656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697542.803601:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134438656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697542.803606:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697542.803608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697542.803610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697542.803614:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697542.803626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697542.803632:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697542/real 1787697542] req@ffff9d85b9d2e300 x1874531134438784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697542.803640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134438784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697542.803647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697542.803650:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697542.803653:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697542.803658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697542.803675:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697542.803685:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697542/real 1787697542] req@ffff9d85b9d2c380 x1874531134438912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697542.803696:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134438912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697542.803701:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697542.803703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697542.803705:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134438912:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697543.379010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697544.403011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697545.427032:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697546.451002:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697547.475012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697547.922987:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697547.922999:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697547.923004:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697547.923014:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5449 00000100:00080000:1.0:1787697547.923025:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697547.923032:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697547.923067:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134439040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697547.923099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697547.923103:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697547.923104:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697547.923108:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5449 00000100:00080000:1.0:1787697547.923112:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697547.923116:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697547.923125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697547.923139:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134439168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697547.923161:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134439040:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697547.923161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697547.923170:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697547.923172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697547.923183:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5449 00000100:00080000:1.0:1787697547.923187:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697547.923191:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697547.923198:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697547.923212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134439296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697547.923221:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697547.923224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697547.923228:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697547.923229:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697547.923236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5449 00000100:00080000:1.0:1787697547.923241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697547.923255:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697547.923260:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697547.923271:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697547/real 1787697547] req@ffff9d85b9d2c700 x1874531134439040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697547.923274:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134439424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697547.923302:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697547.923305:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697547.923310:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134439040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697547.923317:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697547.923324:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5449 00000100:00080000:1.0:1787697547.923331:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697547.923337:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697547.923338:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697547.923346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:1.0:1787697547.923347:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697547.923353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134439040:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697547.923359:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134439552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697547.923366:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697547.923368:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697547.923370:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697547.923372:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5449 00000100:00080000:1.0:1787697547.923374:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697547.923377:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697547.923381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134439168:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787697547.923381:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697547.923391:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134439680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697547.923398:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697547.923400:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697547.923401:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697547.923403:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5449 00000100:00080000:1.0:1787697547.923405:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697547.923408:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697547.923411:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697547.923418:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697547.923421:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134439808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697547.923432:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697547/real 1787697547] req@ffff9d85b9d2ed80 x1874531134439168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697547.923442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134439168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697547.923447:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697547.923450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697547.923453:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134439168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697547.923457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134439296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697547.923470:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697547.923475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697547/real 1787697547] req@ffff9d85b9d2d500 x1874531134439296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697547.923482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134439296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697547.923486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697547.923488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697547.923490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134439296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697547.923493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134439424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697547.923502:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697547.923507:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697547/real 1787697547] req@ffff9d85b9d2e680 x1874531134439424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697547.923513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134439424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697547.923518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697547.923520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697547.923522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134439424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697547.923525:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134439552:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697547.923533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697547.923545:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697547/real 1787697547] req@ffff9d85b9d2dc00 x1874531134439552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697547.923552:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134439552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697547.923557:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697547.923559:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697547.923560:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134439552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697547.923573:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134439680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697547.923583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697547.923588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697547/real 1787697547] req@ffff9d85b9d2f800 x1874531134439680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697547.923594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134439680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697547.923599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697547.923600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697547.923602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134439680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697547.923605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134439808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697547.923614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697547.923618:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697547/real 1787697547] req@ffff9d85b9d2c700 x1874531134439808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697547.923624:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134439808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697547.923636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697547.923640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697547.923642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134439808:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697548.498987:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697549.523026:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697550.547010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697551.570966:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697552.594994:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697553.042950:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697553.042959:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697553.042963:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697553.042972:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5454 00000100:00080000:2.0:1787697553.042981:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697553.042988:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697553.043022:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134439936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697553.043051:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697553.043054:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697553.043055:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697553.043059:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5454 00000100:00080000:2.0:1787697553.043062:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697553.043065:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697553.043073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697553.043086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134440064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697553.043096:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134439936:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697553.043099:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697553.043103:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697553.043105:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697553.043108:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5454 00000100:00080000:2.0:1787697553.043111:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697553.043114:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697553.043119:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697553.043131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134440192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697553.043140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697553.043143:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697553.043145:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697553.043149:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5454 00000100:00080000:2.0:1787697553.043153:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697553.043159:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697553.043159:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697553.043166:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697553.043178:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134440320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697553.043186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697553.043189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697553.043191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697553.043195:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697553/real 1787697553] req@ffff9d85babd4700 x1874531134439936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697553.043199:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5454 00000100:00080000:2.0:1787697553.043205:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697553.043211:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697553.043215:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134439936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697553.043217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697553.043228:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697553.043232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134440448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697553.043235:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697553.043241:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134439936:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697553.043251:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697553.043255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697553.043256:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697553.043261:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5454 00000100:00100000:0.0:1787697553.043263:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440064:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697553.043266:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697553.043272:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697553.043277:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697553.043284:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697553.043289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134440576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697553.043295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697553/real 1787697553] req@ffff9d85babd5f80 x1874531134440064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697553.043306:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697553.043309:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697553.043313:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697553.043321:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5454 00000100:00100000:0.0:1787697553.043324:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134440064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697553.043326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697553.043334:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697553.043335:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697553.043341:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697553.043342:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697553.043345:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697553.043351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787697553.043358:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134440704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697553.043366:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697553.043377:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697553/real 1787697553] req@ffff9d85babd4e00 x1874531134440192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697553.043385:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134440192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697553.043390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697553.043393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697553.043395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697553.043398:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697553.043406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697553.043411:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697553/real 1787697553] req@ffff9d85babd7100 x1874531134440320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697553.043417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134440320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697553.043422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697553.043423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697553.043425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697553.043428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134440448:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697553.043436:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697553.043440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697553/real 1787697553] req@ffff9d85babd7b80 x1874531134440448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697553.043446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134440448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697553.043450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697553.043451:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697553.043453:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134440448:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697553.043463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134440576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697553.043471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697553.043476:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697553/real 1787697553] req@ffff9d85babd4a80 x1874531134440576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697553.043481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134440576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697553.043485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697553.043487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697553.043489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134440576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697553.043492:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134440704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697553.043499:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697553.043504:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697553/real 1787697553] req@ffff9d85babd5180 x1874531134440704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697553.043509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134440704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697553.043514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697553.043515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697553.043517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134440704:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697553.619010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697554.643018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697555.667006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697556.691007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697557.715049:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697558.162976:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697558.162986:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697558.162991:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697558.163001:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5459 00000100:00080000:3.0:1787697558.163011:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697558.163017:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697558.163054:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134440832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697558.163083:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697558.163086:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697558.163088:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697558.163091:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5459 00000100:00080000:3.0:1787697558.163095:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697558.163098:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697558.163106:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697558.163117:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134440960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697558.163130:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697558.163134:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697558.163136:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697558.163139:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5459 00000100:00080000:3.0:1787697558.163143:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697558.163146:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697558.163150:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697558.163158:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134441088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697558.163170:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697558.163173:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697558.163175:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697558.163178:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5459 00000100:00080000:3.0:1787697558.163181:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697558.163183:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697558.163187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697558.163197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134441216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697558.163204:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697558.163207:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697558.163208:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697558.163211:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5459 00000100:00080000:3.0:1787697558.163219:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697558.163223:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697558.163227:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697558.163234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134441344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697558.163242:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697558.163244:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697558.163246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697558.163248:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5459 00000100:00080000:3.0:1787697558.163252:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697558.163255:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697558.163259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697558.163267:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134441472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697558.163274:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697558.163277:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697558.163279:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697558.163282:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5459 00000100:00080000:3.0:1787697558.163284:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697558.163287:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697558.163291:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697558.163299:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134441600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697558.163376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134440832:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787697558.163431:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697558.163467:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697558/real 1787697558] req@ffff9d85c0fb1f80 x1874531134440832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697558.163482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134440832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697558.163492:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697558.163495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697558.163499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134440832:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697558.163505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440960:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697558.163523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697558.163532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697558/real 1787697558] req@ffff9d85c0fb0000 x1874531134440960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697558.163541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134440960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697558.163548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697558.163563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697558.163567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134440960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697558.163573:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697558.163589:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697558.163598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697558/real 1787697558] req@ffff9d85c0fb0700 x1874531134441088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697558.163608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134441088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697558.163615:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697558.163618:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697558.163621:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697558.163625:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697558.163639:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697558.163647:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697558/real 1787697558] req@ffff9d85c0fb1500 x1874531134441216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697558.163656:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134441216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697558.163663:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697558.163665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697558.163669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697558.163675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134441344:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697558.163688:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697558.163696:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697558/real 1787697558] req@ffff9d85c0fb1880 x1874531134441344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697558.163705:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134441344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697558.163711:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697558.163714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697558.163717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134441344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697558.163721:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134441472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697558.163735:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697558.163742:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697558/real 1787697558] req@ffff9d85c0fb2300 x1874531134441472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697558.163751:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134441472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697558.163757:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697558.163760:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697558.163763:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134441472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697558.163768:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134441600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697558.163781:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697558.163790:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697558/real 1787697558] req@ffff9d85c0fb3480 x1874531134441600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697558.163799:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134441600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697558.163805:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697558.163808:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697558.163811:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134441600:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697558.739029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697559.763079:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697560.786972:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697561.811048:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697562.835025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697563.283005:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697563.283015:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697563.283019:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697563.283028:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5465 00000100:00080000:2.0:1787697563.283038:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697563.283044:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697563.283079:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134441728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697563.283104:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697563.283107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697563.283109:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697563.283113:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5465 00000100:00080000:2.0:1787697563.283122:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697563.283126:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697563.283135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697563.283149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134441856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697563.283160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697563.283165:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134441728:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697563.283165:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697563.283168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697563.283176:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5465 00000100:00080000:2.0:1787697563.283180:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697563.283183:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697563.283188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697563.283200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134441984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697563.283209:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697563.283213:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697563.283215:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697563.283218:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5465 00000100:00080000:2.0:1787697563.283222:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697563.283225:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697563.283229:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697563.283233:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697563.283243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134442112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697563.283253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697563.283256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697563.283257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697563.283261:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5465 00000100:00080000:2.0:1787697563.283264:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697563.283267:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697563.283271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697563.283278:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697563/real 1787697563] req@ffff9d85babd7480 x1874531134441728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697563.283281:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134442240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697563.283309:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697563.283312:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697563.283314:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697563.283317:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134441728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697563.283320:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5465 00000100:00080000:2.0:1787697563.283326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697563.283334:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697563.283336:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787697563.283339:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697563.283343:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697563.283349:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134441728:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697563.283355:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134442368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697563.283365:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697563.283368:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697563.283370:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697563.283374:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5465 00000100:00100000:0.0:1787697563.283376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441856:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697563.283378:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697563.283387:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697563.283392:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697563.283405:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134442496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697563.283408:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697563.283426:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697563/real 1787697563] req@ffff9d85babd5c00 x1874531134441856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697563.283441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134441856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697563.283451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697563.283454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697563.283458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697563.283494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697563.283517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697563.283529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697563/real 1787697563] req@ffff9d85babd4000 x1874531134441984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697563.283541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134441984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697563.283548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697563.283551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697563.283554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134441984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697563.283560:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134442112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697563.283579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697563.283590:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697563/real 1787697563] req@ffff9d85babd4380 x1874531134442112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697563.283602:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134442112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697563.283610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697563.283613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697563.283616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134442112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697563.283621:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134442240:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697563.283640:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697563.283651:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697563/real 1787697563] req@ffff9d85babd6300 x1874531134442240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697563.283661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134442240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697563.283668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697563.283671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697563.283674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134442240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697563.283680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134442368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697563.283698:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697563.283708:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697563/real 1787697563] req@ffff9d85babd6a00 x1874531134442368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697563.283719:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134442368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697563.283726:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697563.283729:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697563.283733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134442368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697563.283750:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134442496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697563.283769:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697563.283779:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697563/real 1787697563] req@ffff9d85babd6680 x1874531134442496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697563.283789:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134442496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697563.283797:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697563.283800:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697563.283803:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134442496:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697563.859015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697564.883084:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697565.907005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697566.931033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697567.955002:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697568.402963:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697568.402973:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697568.402978:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697568.402989:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5470 00000100:00080000:2.0:1787697568.402999:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697568.403006:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697568.403048:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134442624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697568.403077:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697568.403080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697568.403082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697568.403086:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5470 00000100:00080000:2.0:1787697568.403089:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697568.403093:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697568.403101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697568.403116:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134442752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697568.403126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697568.403128:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134442624:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697568.403129:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697568.403133:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697568.403139:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5470 00000100:00080000:2.0:1787697568.403142:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697568.403144:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697568.403148:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697568.403156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134442880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697568.403162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697568.403164:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697568.403165:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697568.403167:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5470 00000100:00080000:2.0:1787697568.403168:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697568.403170:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697568.403173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697568.403181:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134443008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697568.403186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697568.403188:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697568.403190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697568.403192:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5470 00000100:00080000:2.0:1787697568.403194:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697568.403196:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697568.403199:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697568.403204:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697568.403208:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134443136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697568.403214:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697568.403217:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697568.403218:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697568.403220:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5470 00000100:00080000:2.0:1787697568.403223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697568.403225:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697568.403228:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697568.403234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134443264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697568.403239:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697568.403241:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697568.403242:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697568.403247:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5470 00000100:00100000:0.0:1787697568.403248:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697568/real 1787697568] req@ffff9d85babd7800 x1874531134442624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697568.403249:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697568.403255:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697568.403263:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697568.403270:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134442624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697568.403278:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134443392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697568.403280:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697568.403286:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697568.403293:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134442624:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697568.403314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134442752:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697568.403332:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697568.403342:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697568/real 1787697568] req@ffff9d85babd5500 x1874531134442752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697568.403350:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134442752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697568.403356:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697568.403358:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697568.403360:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134442752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697568.403363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134442880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697568.403388:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697568.403399:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697568/real 1787697568] req@ffff9d85babd6d80 x1874531134442880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697568.403412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134442880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697568.403420:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697568.403424:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697568.403427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134442880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697568.403431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697568.403445:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697568.403451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697568/real 1787697568] req@ffff9d85babd6680 x1874531134443008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697568.403458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134443008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697568.403463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697568.403465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697568.403467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697568.403469:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134443136:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697568.403478:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697568.403483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697568/real 1787697568] req@ffff9d85babd6a00 x1874531134443136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697568.403489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134443136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697568.403494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697568.403495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697568.403497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134443136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697568.403499:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134443264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697568.403508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697568.403512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697568/real 1787697568] req@ffff9d85babd6300 x1874531134443264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697568.403518:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134443264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697568.403522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697568.403524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697568.403526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134443264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697568.403529:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134443392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697568.403537:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697568.403541:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697568/real 1787697568] req@ffff9d85babd4380 x1874531134443392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697568.403547:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134443392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697568.403552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697568.403553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697568.403555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134443392:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697568.979030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697570.003044:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697571.027071:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697572.051050:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697573.075060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697573.523010:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697573.523024:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697573.523030:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697573.523043:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5475 00000100:00080000:2.0:1787697573.523058:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697573.523065:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697573.523114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134443520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697573.523149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697573.523153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697573.523155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697573.523160:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5475 00000100:00080000:2.0:1787697573.523164:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697573.523169:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697573.523180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697573.523194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134443648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697573.523205:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697573.523209:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697573.523213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697573.523214:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134443520:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697573.523219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5475 00000100:00080000:2.0:1787697573.523225:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697573.523229:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697573.523235:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697573.523249:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134443776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697573.523258:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697573.523262:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697573.523263:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697573.523267:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5475 00000100:00080000:2.0:1787697573.523270:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697573.523273:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697573.523278:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697573.523308:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697573.523321:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134443904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697573.523333:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697573.523336:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697573.523338:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697573.523342:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5475 00000100:00080000:2.0:1787697573.523346:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697573.523351:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697573.523357:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697573.523361:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697573/real 1787697573] req@ffff9d85babd4000 x1874531134443520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697573.523372:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134444032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697573.523383:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697573.523386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134443520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697573.523389:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697573.523391:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697573.523399:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5475 00000100:00080000:2.0:1787697573.523406:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697573.523411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697573.523412:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697573.523421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697573.523421:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697573.523429:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134443520:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697573.523436:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134444160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697573.523446:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697573.523449:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697573.523451:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697573.523456:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5475 00000100:00100000:0.0:1787697573.523460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443648:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697573.523460:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697573.523467:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697573.523474:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697573.523498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697573.523504:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134444288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697573.523515:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697573/real 1787697573] req@ffff9d85babd5c00 x1874531134443648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697573.523527:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134443648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697573.523533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697573.523535:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697573.523538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697573.523543:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697573.523572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697573.523584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697573/real 1787697573] req@ffff9d85babd7480 x1874531134443776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697573.523595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134443776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697573.523603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697573.523606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697573.523608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697573.523612:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697573.523626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697573.523632:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697573/real 1787697573] req@ffff9d85babd5180 x1874531134443904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697573.523639:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134443904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697573.523644:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697573.523646:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697573.523648:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134443904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697573.523651:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444032:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697573.523660:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697573.523665:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697573/real 1787697573] req@ffff9d85babd4a80 x1874531134444032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697573.523671:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134444032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697573.523676:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697573.523677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697573.523679:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697573.523681:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697573.523690:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697573.523694:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697573/real 1787697573] req@ffff9d85babd7b80 x1874531134444160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697573.523700:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134444160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697573.523705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697573.523706:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697573.523708:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697573.523720:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697573.523730:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697573.523735:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697573/real 1787697573] req@ffff9d85babd7100 x1874531134444288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697573.523741:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134444288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697573.523746:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697573.523748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697573.523749:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444288:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697574.099062:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697575.122991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697576.147026:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697577.171035:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697578.195013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697578.642990:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697578.642999:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697578.643003:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697578.643012:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5480 00000100:00080000:3.0:1787697578.643022:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697578.643028:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697578.643057:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134444416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697578.643087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697578.643090:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697578.643092:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697578.643097:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5480 00000100:00080000:3.0:1787697578.643101:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697578.643105:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697578.643114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697578.643133:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134444544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697578.643137:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134444416:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697578.643151:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697578.643154:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697578.643156:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697578.643161:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5480 00000100:00080000:3.0:1787697578.643165:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697578.643168:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697578.643174:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697578.643185:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134444672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697578.643194:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697578.643209:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697578.643212:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697578.643216:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5480 00000100:00080000:3.0:1787697578.643220:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697578.643224:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697578.643230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697578.643244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134444800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697578.643257:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134444544:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697578.643264:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697578.643269:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697578.643271:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697578.643275:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5480 00000100:00080000:3.0:1787697578.643280:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697578.643285:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697578.643290:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697578.643296:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134444672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697578.643305:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134444928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697578.643315:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697578.643320:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697578.643321:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134444800:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787697578.643325:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697578.643331:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5480 00000100:00080000:3.0:1787697578.643337:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697578.643341:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697578.643345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444928:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787697578.643348:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697578.643362:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134445056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697578.643379:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697578.643382:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697578.643384:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697578.643391:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5480 00000100:00080000:3.0:1787697578.643395:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697578.643399:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697578.643402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445056:192.168.201.137@tcp:8:kworker.0 00000008:00100000:3.0:1787697578.643407:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697578.643421:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134445184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697578.643451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445184:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697579.219102:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697580.243098:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697581.267059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697581.715284:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:1.0:1787697581.715337:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697581.715357:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697581.715418:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697581.715430:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697581.715437:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697581.715445:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697581.715453:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697581.715460:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787697581.715597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697578/real 1787697581] req@ffff9d85c0fb1c00 x1874531134445184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697581.715634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134445184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697581.715646:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697581.715652:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697581.715656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697581.715665:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697578/real 1787697581] req@ffff9d85c0fb3800 x1874531134445056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697581.715676:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134445056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697581.715684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697581.715687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697581.715691:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697581.715698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697578/real 1787697581] req@ffff9d85c0fb0a80 x1874531134444800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697581.715708:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134444800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697581.715716:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697581.715720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697581.715723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134444800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697581.715731:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697578/real 1787697581] req@ffff9d85c0fb0e00 x1874531134444928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697581.715743:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134444928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697581.715751:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697581.715754:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697581.715757:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134444928:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697581.715764:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697578/real 1787697581] req@ffff9d85c0fb2680 x1874531134444544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697581.715776:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134444544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697581.715783:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697581.715787:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697581.715790:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134444544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697581.715798:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697578/real 1787697581] req@ffff9d85c0fb2a00 x1874531134444672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697581.715810:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134444672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697581.715817:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697581.715820:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697581.715823:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134444672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697581.715830:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697578/real 1787697581] req@ffff9d85c0fb3100 x1874531134444416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697581.715842:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134444416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697581.715881:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697581.715886:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697581.715890:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134444416:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787697582.291077:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697583.314975:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697583.762943:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697583.762953:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697583.762957:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697583.762975:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5485 00000100:00080000:3.0:1787697583.762987:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697583.762994:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697583.763028:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134445312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697583.763056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697583.763064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697583.763066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697583.763070:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5485 00000100:00080000:3.0:1787697583.763074:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697583.763078:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697583.763088:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697583.763103:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134445440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697583.763126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697583.763129:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697583.763131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697583.763135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5485 00000100:00080000:3.0:1787697583.763139:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697583.763142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697583.763149:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134445312:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697583.763149:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697583.763163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134445568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697583.763180:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697583.763183:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697583.763185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697583.763189:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5485 00000100:00080000:3.0:1787697583.763192:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697583.763196:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697583.763200:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697583.763212:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134445696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697583.763217:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697583.763226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697583.763230:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697583.763232:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697583.763236:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5485 00000100:00080000:3.0:1787697583.763239:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697583.763243:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697583.763246:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697583.763255:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134445824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697583.763264:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697583/real 1787697583] req@ffff9d85c0fb2d80 x1874531134445312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697583.763264:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697583.763270:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697583.763273:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697583.763282:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5485 00000100:00080000:3.0:1787697583.763287:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697583.763291:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697583.763296:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134445312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697583.763297:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697583.763313:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697583.763314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134445952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697583.763322:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697583.763330:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697583.763331:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134445312:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697583.763339:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697583.763342:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697583.763346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134445440:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697583.763349:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5485 00000100:00080000:3.0:1787697583.763355:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697583.763360:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697583.763366:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697583.763376:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697583.763379:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134446080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697583.763393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697583/real 1787697583] req@ffff9d85c0fb0380 x1874531134445440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697583.763409:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134445440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697583.763417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697583.763422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697583.763425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134445440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697583.763448:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134445568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697583.763468:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697583.763478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697583/real 1787697583] req@ffff9d85c0fb3b80 x1874531134445568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697583.763490:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134445568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697583.763498:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697583.763501:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697583.764062:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134445568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697583.764072:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134445696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697583.764258:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697583.764277:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697583/real 1787697583] req@ffff9d85c0fb3100 x1874531134445696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697583.764292:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134445696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697583.764302:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697583.764305:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697583.764309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134445696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697583.764315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445824:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697583.764337:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697583.764348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697583/real 1787697583] req@ffff9d85c0fb2a00 x1874531134445824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697583.764360:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134445824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697583.764367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697583.764371:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697583.764374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445824:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697583.764381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697583.764399:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697583.764409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697583/real 1787697583] req@ffff9d85c0fb2680 x1874531134445952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697583.764421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134445952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697583.764428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697583.764432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697583.764435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134445952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697583.764441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697583.764458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697583.764468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697583/real 1787697583] req@ffff9d85c0fb0e00 x1874531134446080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697583.764485:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134446080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697583.764493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697583.764496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697583.764499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446080:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697584.339022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697585.363030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697586.387000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697587.410970:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697588.435058:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697588.882984:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697588.882994:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697588.882998:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697588.883008:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5490 00000100:00080000:3.0:1787697588.883019:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697588.883036:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697588.883073:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134446208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697588.883105:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697588.883109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697588.883110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697588.883115:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5490 00000100:00080000:3.0:1787697588.883118:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697588.883121:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697588.883130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697588.883145:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134446336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697588.883156:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697588.883162:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134446208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697588.883162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697588.883174:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697588.883180:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5490 00000100:00080000:3.0:1787697588.883185:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697588.883189:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697588.883195:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697588.883207:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134446464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697588.883217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697588.883220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697588.883222:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697588.883226:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697588.883230:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5490 00000100:00080000:3.0:1787697588.883235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697588.883239:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697588.883244:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697588.883257:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134446592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697588.883267:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697588.883271:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697588.883275:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697588.883276:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697588/real 1787697588] req@ffff9d85c0fb0a80 x1874531134446208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697588.883281:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5490 00000100:00080000:3.0:1787697588.883287:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697588.883293:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697588.883299:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697588.883303:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134446208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697588.883313:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134446720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697588.883317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697588.883327:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697588.883334:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134446208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697588.883345:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697588.883349:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697588.883352:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697588.883356:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5490 00000100:00080000:3.0:1787697588.883360:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697588.883365:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134446336:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697588.883366:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697588.883380:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697588.883393:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134446848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697588.883400:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697588.883416:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697588/real 1787697588] req@ffff9d85c0fb3800 x1874531134446336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697588.883421:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697588.883430:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697588.883434:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697588.883439:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134446336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697588.883440:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5490 00000100:00080000:3.0:1787697588.883446:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697588.883454:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697588.883462:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697588.883465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697588.883471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697588.883475:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134446976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697588.883477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134446336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697588.883490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134446464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697588.883515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697588.883529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697588/real 1787697588] req@ffff9d85c0fb1c00 x1874531134446464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697588.883554:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134446464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697588.883562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697588.883565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697588.883569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134446464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697588.883575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134446592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697588.883595:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697588.883606:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697588/real 1787697588] req@ffff9d85c0fb3480 x1874531134446592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697588.883628:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134446592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697588.883635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697588.883638:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697588.883642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134446592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697588.883648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446720:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697588.883667:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697588.883678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697588/real 1787697588] req@ffff9d85c0fb2300 x1874531134446720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697588.883699:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134446720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697588.883707:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697588.883710:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697588.883713:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697588.883730:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697588.883749:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697588.883759:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697588/real 1787697588] req@ffff9d85c0fb1880 x1874531134446848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697588.883770:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134446848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697588.883778:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697588.883781:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697588.883784:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697588.883789:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697588.883807:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697588.883819:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697588/real 1787697588] req@ffff9d85c0fb1500 x1874531134446976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697588.883830:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134446976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697588.883837:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697588.883840:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697588.883843:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134446976:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697589.459044:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697590.484321:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697591.507031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697592.531056:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697593.555024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697594.003000:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697594.003008:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697594.003013:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697594.003021:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5495 00000100:00080000:2.0:1787697594.003031:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697594.003038:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697594.003070:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134447104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697594.003092:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697594.003095:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697594.003096:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697594.003099:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5495 00000100:00080000:2.0:1787697594.003101:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697594.003104:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697594.003110:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697594.003119:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134447232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697594.003126:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697594.003128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697594.003130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697594.003134:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134447104:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697594.003135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5495 00000100:00080000:2.0:1787697594.003139:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697594.003144:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697594.003147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697594.003155:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134447360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697594.003160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697594.003163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697594.003164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697594.003166:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5495 00000100:00080000:2.0:1787697594.003168:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697594.003170:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697594.003173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697594.003181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134447488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697594.003186:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697594.003190:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697594.003191:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697594.003194:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697594.003197:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5495 00000100:00080000:2.0:1787697594.003200:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697594.003203:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697594.003206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697594.003213:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134447616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697594.003219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697594.003221:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697594.003222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697594.003227:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697594/real 1787697594] req@ffff9d85babd4e00 x1874531134447104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697594.003228:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5495 00000100:00080000:2.0:1787697594.003232:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697594.003237:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697594.003242:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697594.003250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134447744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697594.003252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134447104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697594.003266:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697594.003269:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697594.003271:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697594.003271:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697594.003278:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5495 00000100:00080000:2.0:1787697594.003280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697594.003282:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697594.003284:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697594.003285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697594.003293:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134447104:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697594.003294:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134447872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697594.003323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134447232:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697594.003347:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697594.003355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697594/real 1787697594] req@ffff9d85babd5f80 x1874531134447232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697594.003363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134447232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697594.003371:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697594.003373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697594.003376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134447232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697594.003379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134447360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697594.003390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697594.003395:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697594/real 1787697594] req@ffff9d85babd4700 x1874531134447360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697594.003401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134447360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697594.003417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697594.003419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697594.003421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134447360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697594.003424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134447488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697594.003434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697594.003440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697594/real 1787697594] req@ffff9d85babd7100 x1874531134447488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697594.003446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134447488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697594.003450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697594.003452:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697594.003454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134447488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697594.003457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134447616:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697594.003466:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697594.003470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697594/real 1787697594] req@ffff9d85babd7b80 x1874531134447616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697594.003477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134447616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697594.003481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697594.003483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697594.003484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134447616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697594.003487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134447744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697594.003495:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697594.003500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697594/real 1787697594] req@ffff9d85babd4a80 x1874531134447744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697594.003506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134447744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697594.003510:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697594.003511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697594.003513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134447744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697594.003516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134447872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697594.003523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697594.003528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697594/real 1787697594] req@ffff9d85babd5180 x1874531134447872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697594.003534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134447872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697594.003539:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697594.003540:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697594.003542:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134447872:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697594.579034:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697595.603310:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697596.627035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697597.651052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697598.675000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697599.122984:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697599.122991:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697599.122994:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697599.123002:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5500 00000100:00080000:1.0:1787697599.123009:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697599.123015:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697599.123047:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134448000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697599.123063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697599.123065:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697599.123066:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697599.123069:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5500 00000100:00080000:1.0:1787697599.123075:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697599.123077:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697599.123086:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697599.123098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134448128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697599.123102:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134448000:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697599.123116:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697599.123120:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697599.123122:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697599.123126:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5500 00000100:00080000:1.0:1787697599.123130:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697599.123135:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697599.123140:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697599.123153:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134448256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697599.123156:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697599.123169:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697599.123172:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697599.123174:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697599.123179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5500 00000100:00080000:1.0:1787697599.123183:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697599.123188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697599.123190:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697599/real 1787697599] req@ffff9d85b9d2c380 x1874531134448000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697599.123195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697599.123211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134448384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697599.123216:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134448000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697599.123234:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697599.123237:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697599.123238:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697599.123241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697599.123244:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5500 00000100:00080000:1.0:1787697599.123247:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697599.123251:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697599.123256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697599.123257:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697599.123263:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134448000:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697599.123269:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134448512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697599.123279:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697599.123282:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697599.123285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697599.123289:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134448128:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697599.123290:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5500 00000100:00080000:1.0:1787697599.123297:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697599.123303:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697599.123308:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697599.123318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697599.123323:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134448640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697599.123330:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697599/real 1787697599] req@ffff9d85b9d2e300 x1874531134448128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697599.123337:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697599.123343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134448128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697599.123344:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697599.123347:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697599.123356:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697599.123362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697599.123366:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5500 00000100:00100000:0.0:1787697599.123369:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134448128:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697599.123371:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697599.123378:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697599.123385:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697599.123390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134448256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697599.123399:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134448768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697599.123417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697599.123430:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697599/real 1787697599] req@ffff9d85b9d2ce00 x1874531134448256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697599.123442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134448256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697599.123449:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697599.123453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697599.123455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134448256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697599.123459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134448384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697599.123473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697599.123480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697599/real 1787697599] req@ffff9d85b9d2c000 x1874531134448384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697599.123486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134448384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697599.123491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697599.123493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697599.123495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134448384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697599.123498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134448512:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697599.123507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697599.123512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697599/real 1787697599] req@ffff9d85b9d2ca80 x1874531134448512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697599.123518:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134448512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697599.123522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697599.123524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697599.123525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134448512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697599.123537:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134448640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697599.123546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697599.123551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697599/real 1787697599] req@ffff9d85b9d2f480 x1874531134448640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697599.123557:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134448640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697599.123561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697599.123563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697599.123564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134448640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697599.123568:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134448768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697599.123582:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697599.123592:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697599/real 1787697599] req@ffff9d85b9d2fb80 x1874531134448768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697599.123604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134448768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697599.123612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697599.123615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697599.123618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134448768:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697599.698992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697600.723071:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697601.747072:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697602.770990:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697603.795000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697604.242969:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697604.242976:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697604.242979:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697604.242986:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5506 00000100:00080000:2.0:1787697604.242995:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697604.243000:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697604.243030:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134448896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697604.243052:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697604.243056:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697604.243058:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697604.243061:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5506 00000100:00080000:2.0:1787697604.243065:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697604.243068:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697604.243076:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697604.243088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134449024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697604.243097:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697604.243100:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697604.243101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134448896:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697604.243102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697604.243105:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5506 00000100:00080000:2.0:1787697604.243108:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697604.243111:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697604.243116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697604.243137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134449152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697604.243145:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697604.243148:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697604.243149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697604.243153:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5506 00000100:00080000:2.0:1787697604.243155:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697604.243158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697604.243162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697604.243165:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697604.243170:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134449280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697604.243177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697604.243180:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697604.243181:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697604.243184:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5506 00000100:00080000:2.0:1787697604.243186:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697604.243189:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697604.243193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697604.243201:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134449408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697604.243208:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697604/real 1787697604] req@ffff9d85babd7480 x1874531134448896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697604.243221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697604.243224:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134448896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697604.243224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697604.243225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697604.243229:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5506 00000100:00080000:2.0:1787697604.243232:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697604.243233:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697604.243235:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697604.243237:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697604.243239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697604.243240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134448896:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697604.243247:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134449536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697604.243253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697604.243256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697604.243257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697604.243260:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449024:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697604.243260:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5506 00000100:00080000:2.0:1787697604.243263:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697604.243266:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697604.243269:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697604.243278:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134449664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697604.243279:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697604.243289:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697604/real 1787697604] req@ffff9d85babd5c00 x1874531134449024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697604.243299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134449024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697604.243312:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697604.243315:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697604.243318:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697604.243324:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697604.243340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697604.243347:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697604/real 1787697604] req@ffff9d85babd4000 x1874531134449152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697604.243353:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134449152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697604.243358:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697604.243360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697604.243362:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697604.243365:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697604.243374:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697604.243378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697604/real 1787697604] req@ffff9d85babd4380 x1874531134449280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697604.243385:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134449280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697604.243389:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697604.243390:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697604.243392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697604.243395:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134449408:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697604.243403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697604.243408:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697604/real 1787697604] req@ffff9d85babd6300 x1874531134449408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697604.243414:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134449408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697604.243419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697604.243422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697604.243425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134449408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697604.243430:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134449536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697604.243446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697604.243456:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697604/real 1787697604] req@ffff9d85babd6a00 x1874531134449536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697604.243468:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134449536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697604.243474:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697604.243476:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697604.243478:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134449536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697604.243489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134449664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697604.243501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697604.243508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697604/real 1787697604] req@ffff9d85babd6680 x1874531134449664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697604.243514:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134449664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697604.243518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697604.243520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697604.243522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134449664:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697604.819009:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697605.843045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697606.866979:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697607.891020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697608.914997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697609.362973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697609.362982:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697609.362987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697609.362997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5511 00000100:00080000:2.0:1787697609.363008:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697609.363015:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697609.363053:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134449792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697609.363080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697609.363084:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697609.363086:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697609.363090:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5511 00000100:00080000:2.0:1787697609.363095:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697609.363098:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697609.363105:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697609.363115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134449920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697609.363122:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697609.363125:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134449792:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697609.363125:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697609.363129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697609.363135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5511 00000100:00080000:2.0:1787697609.363137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697609.363138:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697609.363142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697609.363149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134450048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697609.363159:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697609.363161:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697609.363162:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697609.363164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5511 00000100:00080000:2.0:1787697609.363166:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697609.363168:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697609.363171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697609.363177:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134450176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697609.363182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697609.363184:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697609.363186:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697609.363188:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5511 00000100:00080000:2.0:1787697609.363190:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697609.363192:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697609.363195:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697609.363199:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697609.363203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134450304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697609.363209:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697609.363211:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697609.363213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697609.363215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5511 00000100:00080000:2.0:1787697609.363218:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697609.363221:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697609.363223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697609.363229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134450432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697609.363234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697609.363235:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697609.363236:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697609.363239:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5511 00000100:00080000:2.0:1787697609.363242:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697609.363248:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697609.363249:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697609/real 1787697609] req@ffff9d85babd6d80 x1874531134449792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697609.363254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697609.363271:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134450560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697609.363276:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134449792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697609.363293:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697609.363298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697609.363302:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134449792:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697609.363323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449920:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697609.363353:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697609.363372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697609/real 1787697609] req@ffff9d85babd5500 x1874531134449920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697609.363384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134449920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697609.363399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697609.363402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697609.363405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134449920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697609.363411:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697609.363431:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697609.363441:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697609/real 1787697609] req@ffff9d85babd7800 x1874531134450048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697609.363452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134450048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697609.363460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697609.363462:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697609.363466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697609.363471:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697609.363486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697609.363494:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697609/real 1787697609] req@ffff9d85babd6680 x1874531134450176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697609.363504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134450176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697609.363511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697609.363513:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697609.363516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697609.363521:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134450304:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697609.363533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697609.363540:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697609/real 1787697609] req@ffff9d85babd6a00 x1874531134450304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697609.363550:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134450304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697609.363557:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697609.363560:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697609.363562:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134450304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697609.363567:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134450432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697609.363579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697609.363586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697609/real 1787697609] req@ffff9d85babd6300 x1874531134450432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697609.363597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134450432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697609.363604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697609.363606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697609.363609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134450432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697609.363613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134450560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697609.363625:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697609.363632:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697609/real 1787697609] req@ffff9d85babd4380 x1874531134450560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697609.363642:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134450560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697609.363649:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697609.363651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697609.363653:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134450560:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697609.939021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697610.963002:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697611.986967:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697613.010993:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697614.035078:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697614.482963:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697614.482975:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697614.482979:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697614.482988:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5516 00000100:00080000:2.0:1787697614.482998:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697614.483005:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697614.483052:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134450688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697614.483079:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697614.483082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697614.483084:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697614.483087:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5516 00000100:00080000:2.0:1787697614.483091:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697614.483095:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697614.483103:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697614.483115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134450816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697614.483124:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697614.483128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697614.483129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697614.483133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5516 00000100:00080000:2.0:1787697614.483136:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697614.483139:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483145:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134450688:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787697614.483145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697614.483159:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134450944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697614.483170:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697614.483173:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697614.483175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697614.483178:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5516 00000100:00080000:2.0:1787697614.483183:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697614.483187:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697614.483192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697614.483203:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134451072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697614.483212:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697614.483214:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697614.483218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697614.483221:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697614.483226:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5516 00000100:00080000:2.0:1787697614.483230:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697614.483252:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697614.483257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697614.483268:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697614/real 1787697614] req@ffff9d85babd4000 x1874531134450688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697614.483271:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134451200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697614.483295:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697614.483298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697614.483301:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134450688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697614.483302:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697614.483315:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697614.483319:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5516 00000100:00080000:0.0:1787697614.483323:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697614.483326:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697614.483329:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134450688:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697614.483335:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450816:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787697614.483342:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697614.483360:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134451328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697614.483372:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697614.483373:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697614.483375:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697614.483378:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697614.483382:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5516 00000100:00080000:2.0:1787697614.483385:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697614.483389:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697614/real 1787697614] req@ffff9d85babd5c00 x1874531134450816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697614.483394:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697614.483411:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134451456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697614.483415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134450816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697614.483441:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697614.483444:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697614.483448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697614.483473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697614.483500:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697614/real 1787697614] req@ffff9d85babd7480 x1874531134450944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697614.483523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134450944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697614.483531:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697614.483534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697614.483537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134450944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697614.483543:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697614.483562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697614/real 1787697614] req@ffff9d85babd5180 x1874531134451072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697614.483612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134451072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697614.483620:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697614.483623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697614.483627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697614.483633:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134451200:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697614.483649:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483659:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697614/real 1787697614] req@ffff9d85babd4a80 x1874531134451200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697614.483669:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134451200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697614.483676:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697614.483679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697614.483682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134451200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697614.483688:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134451328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697614.483707:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483718:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697614/real 1787697614] req@ffff9d85babd7b80 x1874531134451328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697614.483729:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134451328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697614.483736:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697614.483739:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697614.483742:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134451328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697614.483748:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134451456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697614.483761:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697614.483768:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697614/real 1787697614] req@ffff9d85babd7100 x1874531134451456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697614.483774:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134451456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697614.483779:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697614.483780:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697614.483782:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134451456:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697615.059049:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697616.083013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697617.107036:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697618.131024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697619.154942:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697619.602958:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697619.602966:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697619.602970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697619.602980:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5521 00000100:00080000:1.0:1787697619.602989:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697619.602996:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697619.603033:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134451584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697619.603059:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697619.603063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697619.603065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697619.603069:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5521 00000100:00080000:1.0:1787697619.603072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697619.603076:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697619.603085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697619.603098:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134451584:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697619.603098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134451712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697619.603112:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697619.603115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697619.603117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697619.603121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5521 00000100:00080000:1.0:1787697619.603125:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697619.603130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697619.603135:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697619.603149:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697619.603162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134451840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697619.603172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697619.603176:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697619.603177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697619.603180:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697619/real 1787697619] req@ffff9d85b9d2fb80 x1874531134451584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697619.603184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5521 00000100:00080000:1.0:1787697619.603192:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697619.603198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697619.603203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697619.603208:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134451584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697619.603216:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134451968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697619.603223:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697619.603229:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697619.603232:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697619.603233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697619.603237:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697619.603240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134451584:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697619.603243:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5521 00000100:00080000:1.0:1787697619.603248:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697619.603253:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697619.603256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697619.603265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134452096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697619.603266:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451712:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697619.603280:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697619.603284:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697619.603287:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697619.603290:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697619.603295:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5521 00000100:00080000:1.0:1787697619.603301:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697619.603302:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697619/real 1787697619] req@ffff9d85b9d2f480 x1874531134451712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697619.603309:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697619.603316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134451712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697619.603317:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697619.603332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697619.603335:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134452224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697619.603336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697619.603342:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697619.603351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451840:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787697619.603359:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697619.603363:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697619.603365:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697619.603368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697619.603373:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5521 00000100:00100000:0.0:1787697619.603378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697619/real 1787697619] req@ffff9d85b9d2ca80 x1874531134451840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697619.603380:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697619.603387:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697619.603391:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134451840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697619.603391:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697619.603401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.603404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697619.603409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697619.603410:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134452352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697619.603415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697619.603442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697619.603454:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697619/real 1787697619] req@ffff9d85b9d2c000 x1874531134451968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.603465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134451968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.603472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.603475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697619.603478:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134451968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697619.603504:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452096:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697619.603523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697619.603531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697619/real 1787697619] req@ffff9d85b9d2ce00 x1874531134452096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.603538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134452096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.603543:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.603544:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697619.603546:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697619.603549:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697619.603560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697619.603569:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697619/real 1787697619] req@ffff9d85b9d2e300 x1874531134452224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.603579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134452224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.603585:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.603588:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697619.603590:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697619.603596:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697619.603613:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697619.603623:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697619/real 1787697619] req@ffff9d85b9d2c380 x1874531134452352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.603633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134452352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.603641:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.603643:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697619.603646:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452352:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697620.179021:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697621.203062:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697622.226979:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697623.250999:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697624.274977:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697624.722922:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697624.722932:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697624.722936:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697624.722946:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5526 00000100:00080000:1.0:1787697624.722956:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697624.722963:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697624.722992:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134452480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697624.723013:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697624.723016:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697624.723017:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697624.723020:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5526 00000100:00080000:1.0:1787697624.723022:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697624.723025:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697624.723031:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697624.723038:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134452608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697624.723044:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697624.723047:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697624.723048:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697624.723050:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5526 00000100:00080000:1.0:1787697624.723052:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697624.723055:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697624.723060:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697624.723061:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134452480:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697624.723073:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134452736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697624.723084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697624.723088:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697624.723090:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697624.723094:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5526 00000100:00080000:1.0:1787697624.723109:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697624.723112:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697624.723116:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697624.723119:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697624.723128:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134452864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697624.723135:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697624.723138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697624.723139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697624.723141:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5526 00000100:00080000:1.0:1787697624.723144:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697624.723146:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697624.723149:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697624.723156:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134452992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697624.723161:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697624/real 1787697624] req@ffff9d85b9d2c700 x1874531134452480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697624.723164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697624.723166:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697624.723167:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697624.723172:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5526 00000100:00080000:1.0:1787697624.723176:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697624.723181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697624.723186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697624.723193:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134453120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697624.723197:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134452480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697624.723226:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697624.723232:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697624.723232:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697624.723240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134452480:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697624.723240:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697624.723245:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697624.723251:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134452608:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697624.723252:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5526 00000100:00080000:1.0:1787697624.723260:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697624.723266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697624.723271:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697624.723280:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697624.723287:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134453248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697624.723294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697624/real 1787697624] req@ffff9d85b9d2f800 x1874531134452608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697624.723307:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134452608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697624.723320:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697624.723324:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697624.723327:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134452608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697624.723347:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134452736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697624.723364:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697624.723373:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697624/real 1787697624] req@ffff9d85b9d2dc00 x1874531134452736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697624.723384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134452736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697624.723391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697624.723394:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697624.723397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134452736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697624.723456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134452864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697624.723475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697624.723485:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697624/real 1787697624] req@ffff9d85b9d2e680 x1874531134452864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697624.723495:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134452864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697624.723502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697624.723505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697624.723508:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134452864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697624.723513:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452992:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697624.723527:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697624.723535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697624/real 1787697624] req@ffff9d85b9d2d500 x1874531134452992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697624.723545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134452992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697624.723552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697624.723554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697624.723557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134452992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697624.723563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134453120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697624.723577:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697624.723586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697624/real 1787697624] req@ffff9d85b9d2ed80 x1874531134453120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697624.723595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134453120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697624.723602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697624.723605:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697624.723608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134453120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697624.723612:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134453248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697624.723626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697624.723634:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697624/real 1787697624] req@ffff9d85b9d2c380 x1874531134453248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697624.723644:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134453248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697624.723651:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697624.723654:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697624.723657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134453248:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697625.299004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697626.322952:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697627.347017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697628.371000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697629.394989:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697629.842946:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697629.842954:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697629.842957:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697629.842964:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5531 00000100:00080000:1.0:1787697629.842972:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697629.842978:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697629.843012:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134453376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697629.843038:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697629.843040:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697629.843041:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697629.843044:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5531 00000100:00080000:1.0:1787697629.843047:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697629.843049:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697629.843057:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697629.843068:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134453504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697629.843074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697629.843076:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697629.843077:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697629.843080:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5531 00000100:00080000:1.0:1787697629.843081:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697629.843083:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697629.843086:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697629.843093:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134453376:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697629.843093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134453632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697629.843106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697629.843108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697629.843109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697629.843111:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5531 00000100:00080000:1.0:1787697629.843112:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697629.843114:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697629.843117:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697629.843126:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134453760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697629.843135:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697629.843138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697629.843140:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697629.843144:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5531 00000100:00080000:1.0:1787697629.843155:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697629.843160:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697629.843165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697629.843167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697629.843180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134453888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697629.843188:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697629.843190:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697629.843191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697629.843194:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5531 00000100:00080000:1.0:1787697629.843196:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697629.843199:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697629.843203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697629.843211:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697629/real 1787697629] req@ffff9d85b9d2e300 x1874531134453376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697629.843212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134454016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697629.843229:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697629.843236:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697629.843238:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697629.843240:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5531 00000100:00080000:1.0:1787697629.843243:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697629.843245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697629.843248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697629.843249:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134453376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697629.843259:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134454144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697629.843266:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697629.843272:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697629.843277:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134453376:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697629.843286:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134453504:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697629.843310:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697629.843337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697629/real 1787697629] req@ffff9d85b9d2ce00 x1874531134453504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697629.843348:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134453504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697629.843361:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697629.843364:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697629.843367:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134453504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697629.843392:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134453632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697629.843414:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697629.843425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697629/real 1787697629] req@ffff9d85b9d2c000 x1874531134453632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697629.843437:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134453632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697629.843445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697629.843449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697629.843452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134453632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697629.843458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134453760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697629.843477:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697629.843488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697629/real 1787697629] req@ffff9d85b9d2ca80 x1874531134453760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697629.843500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134453760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697629.843507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697629.843510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697629.843513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134453760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697629.843519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134453888:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697629.843538:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697629.843549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697629/real 1787697629] req@ffff9d85b9d2f480 x1874531134453888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697629.843560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134453888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697629.843567:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697629.843570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697629.843574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134453888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697629.843579:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697629.843598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697629.843610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697629/real 1787697629] req@ffff9d85b9d2fb80 x1874531134454016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697629.843621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134454016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697629.843641:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697629.843645:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697629.843648:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697629.843654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697629.843674:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697629.843685:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697629/real 1787697629] req@ffff9d85b9d2c380 x1874531134454144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697629.843697:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134454144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697629.843704:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697629.843707:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697629.843711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454144:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697630.419019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697631.443011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697632.467024:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697633.491102:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697634.515029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697634.962922:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697634.962929:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697634.962932:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697634.962939:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5536 00000100:00080000:1.0:1787697634.962947:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697634.962952:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697634.962980:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134454272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697634.962999:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697634.963001:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697634.963003:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697634.963005:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5536 00000100:00080000:1.0:1787697634.963007:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697634.963009:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697634.963016:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697634.963027:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134454400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697634.963034:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697634.963036:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697634.963037:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697634.963039:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5536 00000100:00080000:1.0:1787697634.963041:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697634.963044:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697634.963047:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697634.963054:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134454272:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697634.963056:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134454528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697634.963067:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697634.963069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697634.963070:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697634.963072:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5536 00000100:00080000:1.0:1787697634.963073:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697634.963075:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697634.963078:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697634.963087:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134454656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697634.963095:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697634.963098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697634.963099:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697634.963102:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5536 00000100:00080000:1.0:1787697634.963106:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697634.963109:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697634.963114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697634.963117:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697634.963127:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134454784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697634.963137:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697634.963140:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697634.963141:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697634.963144:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5536 00000100:00080000:1.0:1787697634.963147:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697634.963150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697634.963155:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697634.963159:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697634/real 1787697634] req@ffff9d85b9d2ed80 x1874531134454272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697634.963169:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134454912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697634.963179:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134454272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697634.963180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697634.963188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697634.963192:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697634.963200:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5536 00000100:00080000:0.0:1787697634.963201:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697634.963206:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697634.963209:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697634.963214:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697634.963217:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134454272:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787697634.963220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697634.963247:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134455040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697634.963254:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134454400:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697634.963291:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697634.963303:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697634/real 1787697634] req@ffff9d85b9d2d500 x1874531134454400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697634.963314:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134454400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697634.963328:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697634.963331:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697634.963334:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134454400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697634.963357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134454528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697634.963384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697634.963393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697634/real 1787697634] req@ffff9d85b9d2e680 x1874531134454528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697634.963403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134454528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697634.963411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697634.963414:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697634.963417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134454528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697634.963422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134454656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697634.963439:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697634.963448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697634/real 1787697634] req@ffff9d85b9d2dc00 x1874531134454656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697634.963460:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134454656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697634.963467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697634.963469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697634.963472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134454656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697634.963478:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454784:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697634.963494:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697634.963504:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697634/real 1787697634] req@ffff9d85b9d2f800 x1874531134454784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697634.963515:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134454784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697634.963523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697634.963525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697634.963529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454784:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697634.963533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697634.963551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697634.963561:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697634/real 1787697634] req@ffff9d85b9d2c700 x1874531134454912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697634.963571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134454912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697634.963579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697634.963582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697634.963586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134454912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697634.963591:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697634.963609:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697634.963619:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697634/real 1787697634] req@ffff9d85b9d2c380 x1874531134455040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697634.963628:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134455040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697634.963636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697634.963639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697634.963642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455040:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697635.539032:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697636.562988:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697637.587044:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697638.610946:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697639.634995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697640.082970:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697640.082980:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697640.082984:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697640.082994:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5541 00000100:00080000:1.0:1787697640.083004:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697640.083011:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697640.083045:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134455168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697640.083071:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697640.083075:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697640.083077:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697640.083081:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5541 00000100:00080000:1.0:1787697640.083084:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697640.083089:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697640.083098:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697640.083112:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134455168:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697640.083115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134455296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697640.083128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697640.083131:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697640.083133:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697640.083136:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5541 00000100:00080000:1.0:1787697640.083154:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697640.083158:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697640.083164:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697640.083175:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134455424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697640.083184:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697640.083190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697640.083194:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697640.083196:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697640.083201:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5541 00000100:00080000:1.0:1787697640.083206:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697640.083210:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697640.083215:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697640.083226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134455552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697640.083236:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697640/real 1787697640] req@ffff9d85b9d2fb80 x1874531134455168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697640.083237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697640.083246:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697640.083250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697640.083258:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5541 00000100:00080000:1.0:1787697640.083262:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697640.083266:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134455168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697640.083270:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697640.083278:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697640.083283:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697640.083289:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697640.083293:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134455680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697640.083298:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134455168:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697640.083311:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697640.083315:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697640.083317:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697640.083322:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5541 00000100:00100000:0.0:1787697640.083326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134455296:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697640.083327:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697640.083335:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697640.083340:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697640.083349:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697640.083354:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134455808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697640.083361:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697640/real 1787697640] req@ffff9d85b9d2f480 x1874531134455296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697640.083376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134455296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697640.083401:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697640.083403:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697640.083409:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697640.083411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697640.083413:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697640.083419:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134455296:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697640.083421:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5541 00000100:00080000:1.0:1787697640.083428:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697640.083431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134455424:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787697640.083436:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697640.083442:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697640.083451:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697640.083457:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134455936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697640.083469:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697640/real 1787697640] req@ffff9d85b9d2ca80 x1874531134455424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697640.083478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134455424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697640.083483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697640.083486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697640.083489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134455424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697640.083493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134455552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697640.083503:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697640.083508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697640/real 1787697640] req@ffff9d85b9d2c000 x1874531134455552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697640.083515:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134455552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697640.083523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697640.083525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697640.083528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134455552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697640.083534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455680:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697640.083551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697640.083560:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697640/real 1787697640] req@ffff9d85b9d2ce00 x1874531134455680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697640.083568:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134455680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697640.083573:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697640.083574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697640.083576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697640.083588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697640.083605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697640.083615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697640/real 1787697640] req@ffff9d85b9d2e300 x1874531134455808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697640.083627:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134455808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697640.083634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697640.083637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697640.083639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697640.083643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697640.083661:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697640.083668:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697640/real 1787697640] req@ffff9d85b9d2fb80 x1874531134455936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697640.083678:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134455936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697640.083685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697640.083687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697640.083690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134455936:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697640.659013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697641.682988:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697642.707005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697643.730971:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697644.755048:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697645.203033:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697645.203044:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697645.203061:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697645.203073:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5546 00000100:00080000:2.0:1787697645.203084:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697645.203091:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697645.203134:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134456064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697645.203163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697645.203166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697645.203167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697645.203170:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5546 00000100:00080000:2.0:1787697645.203172:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697645.203174:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697645.203183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697645.203192:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134456192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697645.203201:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697645.203205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697645.203207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697645.203212:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5546 00000100:00080000:2.0:1787697645.203215:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697645.203219:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697645.203226:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134456064:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787697645.203227:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697645.203243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134456320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697645.203254:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697645.203257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697645.203259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697645.203262:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5546 00000100:00080000:2.0:1787697645.203266:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697645.203270:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697645.203275:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697645.203285:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134456448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697645.203294:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697645.203297:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697645.203299:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697645.203302:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5546 00000100:00080000:2.0:1787697645.203306:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697645.203310:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697645.203315:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697645.203336:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134456576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697645.203344:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697645.203348:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697645.203350:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697645.203355:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134456192:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697645.203356:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5546 00000100:00080000:2.0:1787697645.203363:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697645.203368:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697645.203373:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697645.203385:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134456704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697645.203393:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697645.203397:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697645.203399:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697645.203403:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5546 00000100:00080000:2.0:1787697645.203407:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697645.203410:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697645.203414:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697645.203420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134456320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787697645.203430:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134456832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697645.203454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134456448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697645.203480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134456576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697645.203506:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134456704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697645.203533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134456832:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697645.779016:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697646.803005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697647.827039:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697648.275105:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:1.0:1787697648.275161:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697648.275177:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697648.275214:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697648.275223:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697648.275231:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697648.275238:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697648.275245:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697648.275251:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787697648.275258:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697645/real 1787697648] req@ffff9d85babd4e00 x1874531134456320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697648.275281:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134456320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697648.275292:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697648.275298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697648.275302:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134456320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697648.275309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697645/real 1787697648] req@ffff9d85babd7100 x1874531134456448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697648.275320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134456448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697648.275340:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697648.275343:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697648.275346:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134456448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697648.275353:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697645/real 1787697648] req@ffff9d85babd7b80 x1874531134456576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697648.275364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134456576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697648.275370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697648.275373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697648.275376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134456576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697648.275394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697645/real 1787697648] req@ffff9d85babd4a80 x1874531134456704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697648.275405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134456704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697648.275413:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697648.275417:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697648.275420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134456704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697648.275426:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697645/real 1787697648] req@ffff9d85babd4700 x1874531134456064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697648.275438:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134456064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697648.275446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697648.275448:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697648.275452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134456064:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697648.275457:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697645/real 1787697648] req@ffff9d85babd5f80 x1874531134456192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697648.275469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134456192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697648.275475:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697648.275478:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697648.275481:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134456192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697648.275511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697645/real 1787697648] req@ffff9d85babd5180 x1874531134456832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697648.275523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134456832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697648.275531:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697648.275534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697648.275537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134456832:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697648.851032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697649.875032:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697650.322946:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697650.322954:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697650.322957:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697650.322978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5552 00000100:00080000:2.0:1787697650.322989:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697650.322999:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697650.323030:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134456960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697650.323054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697650.323057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697650.323059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697650.323063:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5552 00000100:00080000:2.0:1787697650.323067:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697650.323071:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697650.323080:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697650.323093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134457088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697650.323104:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134456960:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697650.323105:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697650.323112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697650.323115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697650.323130:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5552 00000100:00080000:2.0:1787697650.323138:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697650.323142:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697650.323148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697650.323160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134457216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697650.323168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697650.323173:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787697650.323174:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697650.323177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697650.323184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5552 00000100:00080000:2.0:1787697650.323187:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697650.323191:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697650.323196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697650.323207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134457344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697650.323215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697650.323219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697650.323221:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697650.323225:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697650/real 1787697650] req@ffff9d85babd7480 x1874531134456960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697650.323228:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5552 00000100:00080000:2.0:1787697650.323233:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697650.323241:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697650.323246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697650.323252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134456960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697650.323258:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134457472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697650.323271:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697650.323272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697650.323278:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697650.323280:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697650.323281:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697650.323287:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134456960:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697650.323290:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5552 00000100:00080000:2.0:1787697650.323297:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697650.323301:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697650.323306:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697650.323317:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134457600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697650.323323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457088:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697650.323332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697650.323336:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697650.323338:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697650.323348:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5552 00000100:00080000:2.0:1787697650.323352:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697650.323356:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697650.323360:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697650.323367:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697650.323375:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697650/real 1787697650] req@ffff9d85babd5c00 x1874531134457088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697650.323382:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134457728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697650.323392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134457088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697650.323402:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697650.323406:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697650.323409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697650.323421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697650.323444:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697650.323455:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697650/real 1787697650] req@ffff9d85babd4000 x1874531134457216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697650.323465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134457216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697650.323472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697650.323475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697650.323478:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697650.323484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697650.323501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697650.323510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697650/real 1787697650] req@ffff9d85babd4380 x1874531134457344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697650.323519:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134457344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697650.323525:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697650.323528:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697650.323531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697650.323535:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134457472:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697650.323553:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697650.323563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697650/real 1787697650] req@ffff9d85babd6300 x1874531134457472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697650.323577:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134457472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697650.323583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697650.323586:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697650.323588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134457472:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697650.323602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134457600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697650.323620:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697650.323631:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697650/real 1787697650] req@ffff9d85babd6a00 x1874531134457600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697650.323641:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134457600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697650.323647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697650.323650:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697650.323652:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134457600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697650.323658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134457728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697650.323674:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697650.323683:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697650/real 1787697650] req@ffff9d85babd6680 x1874531134457728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697650.323693:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134457728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697650.323704:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697650.323708:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697650.323711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134457728:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697650.899043:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697651.923014:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697652.947026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697653.970990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697654.995002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697655.442985:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697655.442993:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697655.443001:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697655.443009:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5557 00000100:00080000:1.0:1787697655.443018:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697655.443023:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697655.443052:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134457856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697655.443071:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697655.443074:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697655.443075:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697655.443078:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5557 00000100:00080000:1.0:1787697655.443080:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697655.443082:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697655.443089:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697655.443096:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134457984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697655.443102:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697655.443104:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697655.443105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697655.443108:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5557 00000100:00080000:1.0:1787697655.443109:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697655.443111:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697655.443115:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697655.443123:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134457856:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697655.443123:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134458112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697655.443139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697655.443141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697655.443142:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697655.443145:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5557 00000100:00080000:1.0:1787697655.443147:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697655.443149:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697655.443153:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697655.443160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134458240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697655.443165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697655.443167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697655.443168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697655.443170:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5557 00000100:00080000:1.0:1787697655.443174:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697655.443177:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697655.443179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697655.443183:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697655.443187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134458368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697655.443193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697655.443195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697655.443197:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697655.443199:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5557 00000100:00080000:1.0:1787697655.443202:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697655.443204:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697655.443206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697655.443213:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134458496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697655.443219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697655.443221:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697655/real 1787697655] req@ffff9d85b9d2c380 x1874531134457856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697655.443223:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697655.443227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697655.443236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5557 00000100:00080000:1.0:1787697655.443241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697655.443245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697655.443249:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134457856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697655.443259:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697655.443263:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697655.443269:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697655.443275:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134457856:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697655.443275:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134458624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697655.443289:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457984:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697655.443312:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697655.443324:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697655/real 1787697655] req@ffff9d85b9d2c700 x1874531134457984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697655.443334:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134457984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697655.443341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697655.443344:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697655.443348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134457984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697655.443368:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134458112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697655.443385:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697655.443394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697655/real 1787697655] req@ffff9d85b9d2f800 x1874531134458112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697655.443404:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134458112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697655.443412:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697655.443415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697655.443417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134458112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697655.443422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134458240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697655.443439:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697655.443449:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697655/real 1787697655] req@ffff9d85b9d2dc00 x1874531134458240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697655.443459:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134458240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697655.443465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697655.443467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697655.443470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134458240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697655.443474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134458368:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697655.443489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697655.443498:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697655/real 1787697655] req@ffff9d85b9d2e680 x1874531134458368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697655.443509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134458368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697655.443516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697655.443519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697655.443522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134458368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697655.443528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134458496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697655.443546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697655.443556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697655/real 1787697655] req@ffff9d85b9d2d500 x1874531134458496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697655.443567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134458496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697655.443575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697655.443577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697655.443580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134458496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697655.443586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134458624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697655.443603:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697655.443613:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697655/real 1787697655] req@ffff9d85b9d2ed80 x1874531134458624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697655.443623:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134458624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697655.443635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697655.443638:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697655.443641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134458624:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697656.019020:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697657.042989:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697658.066984:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697659.091041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697660.115024:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697660.562988:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697660.562999:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697660.563003:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697660.563013:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5562 00000100:00080000:2.0:1787697660.563025:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697660.563031:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697660.563075:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134458752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697660.563107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697660.563111:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697660.563113:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697660.563117:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5562 00000100:00080000:2.0:1787697660.563121:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697660.563125:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697660.563135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697660.563150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134458880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697660.563156:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134458752:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697660.563162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697660.563167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697660.563169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697660.563173:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5562 00000100:00080000:2.0:1787697660.563177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697660.563182:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697660.563187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697660.563199:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134459008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697660.563207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697660.563210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697660.563212:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697660.563215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5562 00000100:00080000:2.0:1787697660.563219:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697660.563223:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697660.563228:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787697660.563230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697660.563244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134459136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697660.563252:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697660.563255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697660.563256:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697660.563260:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5562 00000100:00080000:2.0:1787697660.563263:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697660.563267:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697660.563272:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697660.563281:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697660/real 1787697660] req@ffff9d85babd7800 x1874531134458752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697660.563285:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134459264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697660.563306:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697660.563309:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697660.563310:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697660.563313:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134458752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697660.563316:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5562 00000100:00080000:2.0:1787697660.563322:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697660.563328:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697660.563331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787697660.563332:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697660.563336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697660.563340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134458752:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697660.563346:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134459392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697660.563357:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697660.563359:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697660.563361:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697660.563363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134458880:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697660.563367:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5562 00000100:00080000:2.0:1787697660.563370:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697660.563372:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697660.563375:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697660.563384:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134459520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697660.563393:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697660.563416:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697660/real 1787697660] req@ffff9d85babd5500 x1874531134458880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697660.563431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134458880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697660.563440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697660.563443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697660.563447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134458880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697660.563453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697660.563475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697660.563486:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697660/real 1787697660] req@ffff9d85babd6d80 x1874531134459008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697660.563497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134459008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697660.563505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697660.563508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697660.563512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697660.563518:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697660.563536:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697660.563546:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697660/real 1787697660] req@ffff9d85babd6680 x1874531134459136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697660.563556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134459136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697660.563564:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697660.563567:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697660.563570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697660.563576:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134459264:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697660.563594:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697660.563605:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697660/real 1787697660] req@ffff9d85babd6a00 x1874531134459264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697660.563616:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134459264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697660.563623:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697660.563626:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697660.563629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134459264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697660.563635:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134459392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697660.563652:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697660.563662:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697660/real 1787697660] req@ffff9d85babd6300 x1874531134459392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697660.563674:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134459392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697660.563681:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697660.563684:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697660.563687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134459392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697660.563705:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134459520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697660.563723:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697660.563732:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697660/real 1787697660] req@ffff9d85babd4380 x1874531134459520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697660.563743:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134459520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697660.563756:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697660.563760:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697660.563762:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134459520:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697661.139032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697662.162981:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697663.186992:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697664.211079:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697665.235012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697665.682934:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697665.682944:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697665.682949:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697665.682959:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5567 00000100:00080000:2.0:1787697665.682971:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697665.682978:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697665.683018:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134459648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697665.683043:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697665.683045:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697665.683047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697665.683050:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5567 00000100:00080000:2.0:1787697665.683053:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697665.683056:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697665.683065:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697665.683073:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134459776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697665.683080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697665.683083:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697665.683084:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697665.683087:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5567 00000100:00080000:2.0:1787697665.683089:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697665.683091:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697665.683094:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697665.683097:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134459648:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697665.683102:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134459904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697665.683110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697665.683112:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697665.683113:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697665.683116:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5567 00000100:00080000:2.0:1787697665.683119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697665.683121:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697665.683123:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697665.683130:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134460032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697665.683141:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697665.683143:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697665.683145:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697665.683147:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5567 00000100:00080000:2.0:1787697665.683149:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697665.683151:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697665.683153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697665.683160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134460160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697665.683165:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697665.683167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697665.683168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697665.683172:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5567 00000800:00000100:0.0:1787697665.683175:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697665.683176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697665.683181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697665.683184:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697665.683190:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134460288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697665.683196:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697665.683199:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697665.683200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697665.683202:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5567 00000100:00080000:2.0:1787697665.683205:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697665.683207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697665.683210:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697665.683218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134460416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697665.683233:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697665/real 1787697665] req@ffff9d85babd4000 x1874531134459648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697665.683253:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134459648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697665.683263:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697665.683268:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697665.683272:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134459648:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697665.683279:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459776:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697665.683301:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697665.683313:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697665/real 1787697665] req@ffff9d85babd5c00 x1874531134459776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697665.683325:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134459776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697665.683333:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697665.683336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697665.683340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697665.683375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697665.683397:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697665.683409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697665/real 1787697665] req@ffff9d85babd7480 x1874531134459904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697665.683420:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134459904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697665.683428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697665.683431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697665.683433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134459904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697665.683438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697665.683456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697665.683466:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697665/real 1787697665] req@ffff9d85babd5180 x1874531134460032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697665.683476:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134460032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697665.683483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697665.683485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697665.683488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697665.683493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134460160:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697665.683512:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697665.683523:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697665/real 1787697665] req@ffff9d85babd5f80 x1874531134460160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697665.683535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134460160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697665.683543:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697665.683546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697665.683550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134460160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697665.683556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134460288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697665.683574:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697665.683585:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697665/real 1787697665] req@ffff9d85babd4700 x1874531134460288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697665.683597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134460288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697665.683604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697665.683607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697665.683611:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134460288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697665.683616:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134460416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697665.683641:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697665.683651:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697665/real 1787697665] req@ffff9d85babd4a80 x1874531134460416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697665.683662:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134460416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697665.683675:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697665.683678:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697665.683681:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134460416:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697666.259056:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697667.282985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697668.307014:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697669.331011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697670.354998:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697670.802975:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697670.802982:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697670.802984:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697670.802994:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5572 00000100:00080000:1.0:1787697670.803005:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697670.803012:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697670.803047:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134460544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697670.803068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697670.803071:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697670.803073:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697670.803076:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5572 00000100:00080000:1.0:1787697670.803079:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697670.803083:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697670.803092:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697670.803106:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134460672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697670.803117:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134460544:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697670.803118:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697670.803123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697670.803124:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697670.803129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5572 00000100:00080000:1.0:1787697670.803135:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697670.803139:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697670.803144:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697670.803165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134460800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697670.803172:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697670.803175:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697670.803176:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697670.803178:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5572 00000100:00080000:1.0:1787697670.803181:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697670.803184:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697670.803187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697670.803192:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697670.803199:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134460928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697670.803206:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697670.803208:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697670.803210:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697670.803212:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5572 00000100:00080000:1.0:1787697670.803214:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697670.803216:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697670.803218:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697670.803224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134461056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697670.803230:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697670.803232:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697670.803235:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697670.803236:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697670/real 1787697670] req@ffff9d85b9d2ed80 x1874531134460544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697670.803239:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5572 00000100:00080000:1.0:1787697670.803242:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697670.803249:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697670.803257:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697670.803264:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134460544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697670.803271:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134461184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697670.803278:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697670.803283:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697670.803286:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697670.803289:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697670.803293:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697670.803294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134460544:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697670.803301:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5572 00000100:00080000:1.0:1787697670.803306:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697670.803310:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697670.803314:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697670.803321:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697670.803327:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134461312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697670.803348:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697670.803363:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697670/real 1787697670] req@ffff9d85b9d2d500 x1874531134460672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697670.803373:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134460672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697670.803380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697670.803383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697670.803386:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697670.803391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697670.803407:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697670.803416:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697670/real 1787697670] req@ffff9d85b9d2e680 x1874531134460800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697670.803426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134460800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697670.803432:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697670.803435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697670.803438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697670.803443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697670.803457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697670.803465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697670/real 1787697670] req@ffff9d85b9d2dc00 x1874531134460928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697670.803475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134460928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697670.803481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697670.803484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697670.803487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134460928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697670.803492:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461056:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697670.803508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697670.803516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697670/real 1787697670] req@ffff9d85b9d2f800 x1874531134461056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697670.803527:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134461056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697670.803534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697670.803537:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697670.803540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697670.803545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697670.803559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697670.803568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697670/real 1787697670] req@ffff9d85b9d2c700 x1874531134461184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697670.803579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134461184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697670.803587:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697670.803589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697670.803592:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697670.803635:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697670.803654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697670.803663:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697670/real 1787697670] req@ffff9d85b9d2c380 x1874531134461312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697670.803673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134461312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697670.803686:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697670.803689:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697670.803692:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461312:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697671.379016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697672.403002:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697673.427012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697674.451010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697675.474962:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697675.922934:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697675.922950:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697675.922954:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697675.922963:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5577 00000100:00080000:1.0:1787697675.922973:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697675.922979:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697675.923012:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134461440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697675.923035:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697675.923037:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697675.923038:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697675.923041:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5577 00000100:00080000:1.0:1787697675.923044:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697675.923047:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697675.923054:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697675.923068:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134461568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697675.923069:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134461440:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697675.923081:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697675.923083:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697675.923084:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697675.923087:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5577 00000100:00080000:1.0:1787697675.923089:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697675.923092:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697675.923095:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697675.923103:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134461696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697675.923108:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697675.923110:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697675.923111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697675.923114:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5577 00000100:00080000:1.0:1787697675.923124:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697675.923131:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697675.923131:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697675.923139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697675.923152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134461824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697675.923161:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697675.923166:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697675/real 1787697675] req@ffff9d85b9d2fb80 x1874531134461440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697675.923166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697675.923170:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697675.923179:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5577 00000100:00080000:1.0:1787697675.923183:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697675.923187:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134461440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697675.923191:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697675.923200:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697675.923201:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697675.923205:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697675.923208:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134461440:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697675.923217:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134461952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697675.923227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134461568:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697675.923228:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697675.923235:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697675.923237:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697675.923241:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5577 00000100:00080000:1.0:1787697675.923244:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697675.923246:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697675.923250:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697675.923255:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697675.923259:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697675/real 1787697675] req@ffff9d85b9d2e300 x1874531134461568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697675.923265:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134462080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697675.923272:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134461568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697675.923274:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697675.923281:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697675.923284:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697675.923291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697675.923291:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5577 00000100:00080000:1.0:1787697675.923296:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697675.923298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697675.923302:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697675.923305:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134461568:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787697675.923307:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697675.923315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134461696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697675.923320:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134462208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697675.923332:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697675.923339:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697675/real 1787697675] req@ffff9d85b9d2ce00 x1874531134461696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697675.923347:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134461696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697675.923352:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697675.923353:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697675.923355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134461696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697675.923358:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134461824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697675.923366:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697675.923370:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697675/real 1787697675] req@ffff9d85b9d2c000 x1874531134461824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697675.923376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134461824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697675.923380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697675.923382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697675.923383:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134461824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697675.923392:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697675.923401:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697675.923406:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697675/real 1787697675] req@ffff9d85b9d2ca80 x1874531134461952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697675.923412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134461952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697675.923416:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697675.923418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697675.923420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134461952:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697675.923422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697675.923430:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697675.923434:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697675/real 1787697675] req@ffff9d85b9d2f480 x1874531134462080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697675.923440:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134462080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697675.923444:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697675.923445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697675.923447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697675.923449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697675.923457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697675.923461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697675/real 1787697675] req@ffff9d85b9d2fb80 x1874531134462208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697675.923467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134462208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697675.923475:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697675.923477:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697675.923478:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462208:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697676.499014:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697677.523001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697678.547048:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697679.571055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697680.595039:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697681.042988:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697681.042998:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697681.043003:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697681.043017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5582 00000100:00080000:1.0:1787697681.043030:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697681.043039:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697681.043085:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134462336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697681.043111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697681.043114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697681.043116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697681.043119:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5582 00000100:00080000:1.0:1787697681.043122:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697681.043125:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697681.043133:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697681.043141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134462464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697681.043148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697681.043151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697681.043152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697681.043155:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5582 00000100:00080000:1.0:1787697681.043156:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697681.043159:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697681.043163:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697681.043175:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134462336:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697681.043177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134462592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697681.043192:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697681.043196:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697681.043198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697681.043202:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5582 00000100:00080000:1.0:1787697681.043206:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697681.043209:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697681.043213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697681.043223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134462720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697681.043230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697681.043232:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697681.043234:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697681.043236:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5582 00000100:00080000:1.0:1787697681.043249:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697681.043251:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697681.043254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697681.043263:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134462848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697681.043264:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697681.043272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697681.043275:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697681.043277:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697681.043279:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5582 00000100:00080000:1.0:1787697681.043281:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697681.043283:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697681.043285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697681.043291:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134462976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697681.043300:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697681.043303:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697681.043304:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697681.043306:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5582 00000100:00080000:1.0:1787697681.043308:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697681.043309:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697681.043317:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697681.043333:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697681/real 1787697681] req@ffff9d85b9d2c380 x1874531134462336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697681.043334:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134463104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697681.043374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134462336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697681.043387:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697681.043392:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697681.043396:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134462336:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697681.043405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134462464:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697681.043442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697681.043458:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697681/real 1787697681] req@ffff9d85b9d2c700 x1874531134462464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697681.043471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134462464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697681.043479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697681.043483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697681.043486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134462464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697681.043512:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134462592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697681.043528:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697681.043536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697681/real 1787697681] req@ffff9d85b9d2f800 x1874531134462592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697681.043543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134462592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697681.043548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697681.043550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697681.043552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134462592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697681.043556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134462720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697681.043565:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697681.043570:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697681/real 1787697681] req@ffff9d85b9d2dc00 x1874531134462720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697681.043576:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134462720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697681.043581:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697681.043582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697681.043584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134462720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697681.043586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462848:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697681.043595:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697681.043601:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697681/real 1787697681] req@ffff9d85b9d2e680 x1874531134462848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697681.043608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134462848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697681.043613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697681.043614:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697681.043616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462848:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697681.043618:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697681.043627:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697681.043632:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697681/real 1787697681] req@ffff9d85b9d2d500 x1874531134462976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697681.043637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134462976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697681.043641:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697681.043643:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697681.043644:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134462976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697681.043647:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134463104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697681.043655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697681.043659:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697681/real 1787697681] req@ffff9d85b9d2ed80 x1874531134463104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697681.043665:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134463104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697681.043673:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697681.043682:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697681.043685:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134463104:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697681.619045:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697682.642990:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697683.667046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697684.690982:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697685.715072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697686.162976:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697686.162987:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697686.162992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697686.163002:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5587 00000100:00080000:2.0:1787697686.163012:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697686.163019:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697686.163060:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134463232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697686.163083:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697686.163086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697686.163087:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697686.163090:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5587 00000100:00080000:2.0:1787697686.163093:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697686.163095:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697686.163104:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697686.163113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134463360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697686.163121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134463232:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697686.163122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697686.163128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697686.163130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697686.163133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5587 00000100:00080000:2.0:1787697686.163135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697686.163137:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697686.163140:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697686.163148:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134463488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697686.163154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697686.163157:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697686.163158:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697686.163160:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5587 00000100:00080000:2.0:1787697686.163162:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697686.163164:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697686.163166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697686.163173:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134463616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697686.163179:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697686.163181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697686.163182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697686.163185:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5587 00000100:00080000:2.0:1787697686.163187:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697686.163191:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697686.163192:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697686.163197:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697686.163204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134463744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697686.163211:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697686.163213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697686.163214:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697686.163216:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5587 00000100:00080000:2.0:1787697686.163218:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697686.163220:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697686.163223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697686.163230:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134463872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697686.163236:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697686/real 1787697686] req@ffff9d85babd7b80 x1874531134463232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697686.163237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697686.163244:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697686.163246:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697686.163253:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5587 00000100:00080000:2.0:1787697686.163256:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697686.163258:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697686.163261:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134463232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697686.163269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697686.163270:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697686.163275:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697686.163282:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134463232:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697686.163282:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134464000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697686.163308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134463360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697686.163324:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697686.163332:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697686/real 1787697686] req@ffff9d85babd7100 x1874531134463360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697686.163340:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134463360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697686.163360:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697686.163362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697686.163364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134463360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697686.163369:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134463488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697686.163384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697686.163390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697686/real 1787697686] req@ffff9d85babd4e00 x1874531134463488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697686.163398:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134463488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697686.163402:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697686.163404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697686.163406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134463488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697686.163409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134463616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697686.163418:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697686.163424:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697686/real 1787697686] req@ffff9d85babd4a80 x1874531134463616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697686.163435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134463616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697686.163442:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697686.163445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697686.163448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134463616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697686.163454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134463744:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697686.163473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697686.163484:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697686/real 1787697686] req@ffff9d85babd4700 x1874531134463744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697686.163496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134463744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697686.163504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697686.163507:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697686.163511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134463744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697686.163516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134463872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697686.163530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697686.163537:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697686/real 1787697686] req@ffff9d85babd5f80 x1874531134463872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697686.163544:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134463872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697686.163549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697686.163551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697686.163552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134463872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697686.163555:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697686.163565:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697686.163573:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697686/real 1787697686] req@ffff9d85babd5180 x1874531134464000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697686.163599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134464000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697686.163613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697686.163616:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697686.163619:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464000:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697686.739065:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697687.763081:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697688.787067:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697689.811064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697690.835046:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697691.282969:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697691.282980:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697691.282985:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697691.282996:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5593 00000100:00080000:2.0:1787697691.283008:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697691.283016:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697691.283059:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134464128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697691.283084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697691.283086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697691.283088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697691.283091:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5593 00000100:00080000:2.0:1787697691.283093:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697691.283095:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697691.283103:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697691.283112:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134464256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697691.283119:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697691.283122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697691.283125:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697691.283126:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134464128:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697691.283128:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5593 00000100:00080000:2.0:1787697691.283130:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697691.283136:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697691.283141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697691.283149:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134464384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697691.283156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697691.283158:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697691.283159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697691.283162:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5593 00000100:00080000:2.0:1787697691.283163:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697691.283165:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697691.283168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697691.283174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134464512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697691.283180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697691.283182:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697691.283183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697691.283186:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5593 00000100:00080000:2.0:1787697691.283188:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697691.283190:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697691.283193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697691.283198:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697691.283201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134464640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697691.283209:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697691.283212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697691.283213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697691.283216:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5593 00000100:00080000:2.0:1787697691.283219:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697691.283222:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697691.283224:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697691.283230:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134464768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697691.283236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697691.283238:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697691.283239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697691.283243:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5593 00000100:00100000:0.0:1787697691.283244:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697691/real 1787697691] req@ffff9d85babd7480 x1874531134464128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697691.283246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697691.283251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697691.283268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697691.283279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134464896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697691.283283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134464128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697691.283297:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697691.283300:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697691.283304:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134464128:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697691.283325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134464256:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697691.283341:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697691.283349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697691/real 1787697691] req@ffff9d85babd5c00 x1874531134464256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697691.283356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134464256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697691.283362:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697691.283364:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697691.283366:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134464256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697691.283369:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134464384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697691.283381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697691.283390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697691/real 1787697691] req@ffff9d85babd4000 x1874531134464384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697691.283402:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134464384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697691.283410:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697691.283413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697691.283416:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134464384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697691.283421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134464512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697691.283441:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697691.283452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697691/real 1787697691] req@ffff9d85babd4380 x1874531134464512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697691.283460:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134464512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697691.283465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697691.283467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697691.283469:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134464512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697691.283473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464640:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697691.283486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697691.283492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697691/real 1787697691] req@ffff9d85babd6300 x1874531134464640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697691.283500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134464640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697691.283504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697691.283506:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697691.283507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697691.283510:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697691.283519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697691.283524:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697691/real 1787697691] req@ffff9d85babd6a00 x1874531134464768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697691.283530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134464768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697691.283535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697691.283536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697691.283538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697691.283540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697691.283552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697691.283561:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697691/real 1787697691] req@ffff9d85babd6680 x1874531134464896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697691.283571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134464896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697691.283584:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697691.283587:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697691.283591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134464896:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697691.859084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697692.883050:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697693.907044:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697694.931027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697695.955093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697696.402991:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697696.403003:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697696.403008:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697696.403027:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5598 00000100:00080000:2.0:1787697696.403040:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697696.403048:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697696.403089:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134465024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697696.403118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697696.403122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697696.403124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697696.403128:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5598 00000100:00080000:2.0:1787697696.403132:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697696.403136:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697696.403147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697696.403163:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134465152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697696.403188:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697696.403189:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134465024:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697696.403195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697696.403198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697696.403202:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5598 00000100:00080000:2.0:1787697696.403205:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697696.403208:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697696.403213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697696.403222:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134465280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697696.403228:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697696.403231:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697696.403232:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697696.403234:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5598 00000100:00080000:2.0:1787697696.403237:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697696.403239:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697696.403242:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697696.403250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134465408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697696.403256:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697696.403258:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697696.403259:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697696.403263:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697696.403263:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5598 00000100:00080000:2.0:1787697696.403268:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697696.403271:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697696.403274:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697696.403282:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134465536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697696.403287:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697696.403290:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697696.403291:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697696.403294:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5598 00000100:00080000:2.0:1787697696.403296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697696.403298:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697696.403301:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697696.403307:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697696/real 1787697696] req@ffff9d85babd6d80 x1874531134465024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697696.403310:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134465664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697696.403334:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697696.403336:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134465024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697696.403338:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697696.403341:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697696.403348:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5598 00000100:00080000:2.0:1787697696.403351:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697696.403354:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697696.403355:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787697696.403358:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697696.403360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697696.403365:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134465024:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697696.403369:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134465792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697696.403389:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134465152:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697696.403404:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697696.403412:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697696/real 1787697696] req@ffff9d85babd5500 x1874531134465152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697696.403419:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134465152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697696.403424:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697696.403426:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697696.403428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134465152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697696.403432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134465280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697696.403442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697696.403448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697696/real 1787697696] req@ffff9d85babd7800 x1874531134465280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697696.403454:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134465280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697696.403459:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697696.403460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697696.403462:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134465280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697696.403464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134465408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697696.403473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697696.403477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697696/real 1787697696] req@ffff9d85babd6680 x1874531134465408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697696.403483:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134465408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697696.403487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697696.403489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697696.403491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134465408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697696.403494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134465536:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697696.403503:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697696.403508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697696/real 1787697696] req@ffff9d85babd6a00 x1874531134465536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697696.403514:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134465536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697696.403519:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697696.403520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697696.403522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134465536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697696.403524:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134465664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697696.403533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697696.403537:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697696/real 1787697696] req@ffff9d85babd6300 x1874531134465664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697696.403543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134465664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697696.403547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697696.403549:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697696.403551:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134465664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697696.403554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134465792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697696.403561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697696.403566:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697696/real 1787697696] req@ffff9d85babd4380 x1874531134465792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697696.403575:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134465792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697696.403588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697696.403590:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697696.403593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134465792:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697696.978983:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697698.003025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697699.026998:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697700.050996:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697701.075065:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697701.523010:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697701.523026:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697701.523030:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697701.523042:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5603 00000100:00080000:1.0:1787697701.523054:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697701.523061:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697701.523121:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134465920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697701.523153:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697701.523155:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697701.523157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697701.523161:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5603 00000100:00080000:1.0:1787697701.523164:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697701.523167:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697701.523178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697701.523193:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134466048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697701.523203:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134465920:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697701.523207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697701.523212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697701.523214:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697701.523219:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5603 00000100:00080000:1.0:1787697701.523223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697701.523228:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697701.523234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697701.523246:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134466176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697701.523253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697701.523256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697701.523257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697701.523260:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5603 00000100:00080000:1.0:1787697701.523263:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697701.523265:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697701.523268:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697701.523272:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697701.523280:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134466304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697701.523288:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697701.523290:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697701.523292:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697701.523295:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5603 00000100:00080000:1.0:1787697701.523297:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697701.523299:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697701.523301:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697701.523308:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134466432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697701.523313:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697701.523315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697701.523317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697701.523320:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697701/real 1787697701] req@ffff9d85b9d2ed80 x1874531134465920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697701.523321:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5603 00000100:00080000:1.0:1787697701.523326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697701.523331:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697701.523337:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697701.523348:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134465920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697701.523355:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134466560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697701.523358:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697701.523366:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697701.523366:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697701.523372:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697701.523374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134465920:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697701.523375:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697701.523380:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5603 00000100:00080000:1.0:1787697701.523382:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697701.523384:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697701.523386:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697701.523395:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134466688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697701.523412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466048:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697701.523429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697701.523438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697701/real 1787697701] req@ffff9d85b9d2d500 x1874531134466048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697701.523446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134466048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697701.523451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697701.523454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697701.523456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466048:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697701.523460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697701.523471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697701.523477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697701/real 1787697701] req@ffff9d85b9d2e680 x1874531134466176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697701.523483:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134466176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697701.523488:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697701.523490:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697701.523491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697701.523495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697701.523504:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697701.523509:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697701/real 1787697701] req@ffff9d85b9d2dc00 x1874531134466304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697701.523515:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134466304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697701.523528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697701.523531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697701.523534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697701.523539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134466432:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697701.523559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697701.523569:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697701/real 1787697701] req@ffff9d85b9d2f800 x1874531134466432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697701.523578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134466432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697701.523590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697701.523592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697701.523594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134466432:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697701.523598:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134466560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697701.523611:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697701.523617:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697701/real 1787697701] req@ffff9d85b9d2c700 x1874531134466560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697701.523624:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134466560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697701.523628:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697701.523630:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697701.523632:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134466560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697701.523635:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134466688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697701.523643:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697701.523648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697701/real 1787697701] req@ffff9d85b9d2c380 x1874531134466688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697701.523654:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134466688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697701.523662:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697701.523664:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697701.523666:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134466688:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697702.099038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697703.123048:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697704.147006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697705.171075:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697706.194949:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697706.642941:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697706.642949:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697706.642952:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697706.642960:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5608 00000100:00080000:1.0:1787697706.642969:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697706.642976:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697706.643025:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134466816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697706.643046:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697706.643049:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697706.643051:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697706.643054:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5608 00000100:00080000:1.0:1787697706.643058:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697706.643062:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697706.643071:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697706.643086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134466944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697706.643088:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134466816:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697706.643102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697706.643106:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697706.643107:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697706.643111:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5608 00000100:00080000:1.0:1787697706.643114:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697706.643117:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697706.643121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697706.643132:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134467072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697706.643141:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697706.643141:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697706.643146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697706.643149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697706.643153:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5608 00000100:00080000:1.0:1787697706.643158:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697706.643163:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697706.643167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697706.643180:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697706/real 1787697706] req@ffff9d85b9d2fb80 x1874531134466816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697706.643182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134467200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697706.643205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697706.643208:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697706.643210:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697706.643211:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134466816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697706.643217:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5608 00000100:00080000:1.0:1787697706.643222:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697706.643227:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697706.643228:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697706.643230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697706.643232:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697706.643237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134466816:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697706.643241:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134467328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697706.643247:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697706.643249:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697706.643250:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697706.643252:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5608 00000100:00080000:1.0:1787697706.643254:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697706.643257:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466944:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697706.643258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697706.643263:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697706.643272:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134467456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697706.643279:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697706.643280:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697706.643283:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697706.643285:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697706.643289:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5608 00000100:00080000:1.0:1787697706.643291:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697706.643293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697706/real 1787697706] req@ffff9d85b9d2f480 x1874531134466944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697706.643296:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697706.643301:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697706.643310:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134466944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697706.643314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134467584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697706.643325:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697706.643329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697706.643333:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134466944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697706.643337:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697706.643349:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697706.643355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697706/real 1787697706] req@ffff9d85b9d2ca80 x1874531134467072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697706.643362:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134467072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697706.643366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697706.643368:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697706.643369:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697706.643372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697706.643381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697706.643386:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697706/real 1787697706] req@ffff9d85b9d2c000 x1874531134467200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697706.643392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134467200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697706.643396:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697706.643397:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697706.643399:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697706.643402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134467328:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697706.643410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697706.643414:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697706/real 1787697706] req@ffff9d85b9d2ce00 x1874531134467328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697706.643421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134467328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697706.643425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697706.643426:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697706.643428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134467328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697706.643438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134467456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697706.643447:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697706.643452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697706/real 1787697706] req@ffff9d85b9d2e300 x1874531134467456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697706.643457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134467456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697706.643462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697706.643463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697706.643464:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134467456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697706.643468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134467584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697706.643475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697706.643480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697706/real 1787697706] req@ffff9d85b9d2fb80 x1874531134467584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697706.643486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134467584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697706.643494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697706.643496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697706.643498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134467584:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697707.219089:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697708.243074:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697709.267065:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697710.290982:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697711.315101:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697711.762984:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697711.762995:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697711.763001:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697711.763014:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5613 00000100:00080000:1.0:1787697711.763028:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697711.763037:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697711.763088:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134467712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697711.763119:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697711.763122:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697711.763124:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697711.763127:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5613 00000100:00080000:1.0:1787697711.763130:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697711.763133:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697711.763143:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697711.763155:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134467840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697711.763168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697711.763172:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697711.763174:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697711.763177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5613 00000100:00080000:1.0:1787697711.763179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697711.763181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697711.763185:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697711.763195:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134467968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697711.763203:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697711.763209:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697711.763211:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134467712:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697711.763211:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697711.763224:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5613 00000100:00080000:1.0:1787697711.763229:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697711.763232:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697711.763236:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697711.763245:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134468096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697711.763250:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697711.763252:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697711.763253:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697711.763256:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5613 00000100:00080000:1.0:1787697711.763259:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697711.763262:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697711.763265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697711.763271:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134468224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697711.763276:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697711.763279:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697711.763280:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697711.763283:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5613 00000100:00080000:1.0:1787697711.763288:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697711.763290:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697711.763292:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697711.763300:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134468352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697711.763309:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697711.763312:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697711.763313:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697711.763315:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5613 00000100:00080000:1.0:1787697711.763317:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697711.763319:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697711.763326:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697711.763335:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134468480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697711.763347:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697711.763396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697711.763423:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697711.763458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134468224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697711.763486:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134468352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697711.763515:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134468480:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697712.339031:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697713.363110:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697714.386987:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787697714.835267:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787697714.835340:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697714.835365:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697714.835431:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697714.835443:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697714.835453:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697714.835463:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697714.835475:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787697714.835484:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787697714.835576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697711/real 1787697714] req@ffff9d85b9d2f800 x1874531134467968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697714.835609:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134467968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697714.835625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697714.835631:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697714.835636:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697714.835647:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697711/real 1787697714] req@ffff9d85b9d2dc00 x1874531134468096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697714.835662:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134468096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697714.835671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697714.835676:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697714.835679:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697714.835687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697711/real 1787697714] req@ffff9d85b9d2e680 x1874531134468224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697714.835701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134468224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697714.835709:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697714.835714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697714.835718:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134468224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697714.835726:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697711/real 1787697714] req@ffff9d85b9d2d500 x1874531134468352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697714.835740:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134468352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697714.835748:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697714.835753:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697714.835757:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134468352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697714.835765:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697711/real 1787697714] req@ffff9d85b9d2ed80 x1874531134468480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697714.835778:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134468480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697714.835787:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697714.835791:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697714.835794:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134468480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697714.835802:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697711/real 1787697714] req@ffff9d85b9d2c380 x1874531134467712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697714.835816:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134467712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697714.835824:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697714.835828:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697714.835832:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134467712:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697714.835839:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697711/real 1787697714] req@ffff9d85b9d2c700 x1874531134467840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697714.835907:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134467840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697714.835917:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697714.835922:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697714.835925:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134467840:192.168.201.137@tcp:38:kworker.0 00010000:00010000:3.0:1787697715.411071:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697716.435092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697716.883091:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697716.883105:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697716.883111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697716.883124:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5618 00000100:00080000:2.0:1787697716.883140:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697716.883150:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697716.883197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134468608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697716.883238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697716.883243:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697716.883246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697716.883251:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5618 00000100:00080000:2.0:1787697716.883255:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697716.883260:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697716.883273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697716.883289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134468736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697716.883300:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697716.883304:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697716.883306:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697716.883310:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5618 00000100:00080000:2.0:1787697716.883314:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697716.883317:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697716.883323:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697716.883332:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134468864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697716.883342:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697716.883346:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697716.883348:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697716.883352:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5618 00000100:00080000:2.0:1787697716.883354:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697716.883358:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697716.883362:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697716.883371:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134468992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697716.883381:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697716.883385:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697716.883387:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697716.883390:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5618 00000100:00080000:2.0:1787697716.883393:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697716.883397:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697716.883414:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697716.883425:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134469120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697716.883435:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697716.883439:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697716.883441:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697716.883444:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5618 00000100:00080000:2.0:1787697716.883447:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697716.883451:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697716.883455:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697716.883466:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134469248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697716.883481:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697716.883485:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697716.883487:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697716.883492:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5618 00000100:00080000:2.0:1787697716.883496:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697716.883500:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697716.883505:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697716.883521:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134469376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697716.883611:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134468608:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787697716.883734:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697716.883875:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697716/real 1787697716] req@ffff9d85babd4000 x1874531134468608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697716.883900:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134468608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697716.883917:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697716.883922:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697716.883927:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134468608:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697716.883935:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468736:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697716.883964:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697716.883979:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697716/real 1787697716] req@ffff9d85babd5c00 x1874531134468736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697716.883992:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134468736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697716.884000:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697716.884004:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697716.884007:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697716.884013:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697716.884033:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697716.884044:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697716/real 1787697716] req@ffff9d85babd7480 x1874531134468864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697716.884056:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134468864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697716.884064:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697716.884068:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697716.884071:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697716.884078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697716.884101:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697716.884114:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697716/real 1787697716] req@ffff9d85babd5180 x1874531134468992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697716.884126:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134468992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697716.884135:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697716.884138:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697716.884141:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134468992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697716.884147:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134469120:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697716.884170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697716.884186:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697716/real 1787697716] req@ffff9d85babd5f80 x1874531134469120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697716.884197:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134469120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697716.884212:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697716.884216:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697716.884220:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134469120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697716.884227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134469248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697716.884250:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697716.884261:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697716/real 1787697716] req@ffff9d85babd4700 x1874531134469248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697716.884272:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134469248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697716.884282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697716.884285:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697716.884288:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134469248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697716.884298:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134469376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697716.884318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697716.884328:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697716/real 1787697716] req@ffff9d85babd4a80 x1874531134469376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697716.884338:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134469376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697716.884347:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697716.884350:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697716.884353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134469376:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697717.459084:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697718.483111:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697719.507048:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697720.531096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697721.555034:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697722.002984:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697722.002993:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697722.002998:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697722.003009:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5623 00000100:00080000:3.0:1787697722.003021:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697722.003028:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697722.003069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134469504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697722.003099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697722.003102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697722.003104:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697722.003107:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5623 00000100:00080000:3.0:1787697722.003109:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697722.003111:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697722.003120:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697722.003128:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134469632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697722.003135:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697722.003138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697722.003139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697722.003142:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5623 00000100:00080000:3.0:1787697722.003143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697722.003148:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697722.003150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134469504:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697722.003152:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697722.003160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134469760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697722.003172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697722.003175:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697722.003176:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697722.003178:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5623 00000100:00080000:3.0:1787697722.003180:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697722.003182:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697722.003185:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697722.003194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134469888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697722.003200:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697722.003203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697722.003204:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697722.003206:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5623 00000100:00080000:3.0:1787697722.003207:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697722.003210:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697722.003212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697722.003217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134470016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697722.003223:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697722.003225:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697722.003227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697722.003229:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5623 00000100:00080000:3.0:1787697722.003232:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697722.003236:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697722.003238:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697722.003243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697722.003248:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134470144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697722.003254:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697722.003256:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697722.003258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697722.003260:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5623 00000100:00080000:3.0:1787697722.003262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697722.003264:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697722.003267:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697722.003278:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134470272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697722.003298:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697722/real 1787697722] req@ffff9d85c0fb0700 x1874531134469504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697722.003314:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134469504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697722.003331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697722.003335:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697722.003340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134469504:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697722.003347:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134469632:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697722.003367:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697722.003376:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697722/real 1787697722] req@ffff9d85c0fb0000 x1874531134469632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697722.003389:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134469632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697722.003394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697722.003396:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697722.003398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134469632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697722.003418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134469760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697722.003430:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697722.003436:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697722/real 1787697722] req@ffff9d85c0fb1f80 x1874531134469760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697722.003443:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134469760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697722.003448:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697722.003450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697722.003452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134469760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697722.003455:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134469888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697722.003465:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697722.003470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697722/real 1787697722] req@ffff9d85c0fb1500 x1874531134469888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697722.003477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134469888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697722.003481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697722.003483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697722.003484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134469888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697722.003487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697722.003495:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697722.003501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697722/real 1787697722] req@ffff9d85c0fb1880 x1874531134470016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697722.003506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134470016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697722.003510:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697722.003512:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697722.003513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697722.003516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697722.003524:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697722.003529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697722/real 1787697722] req@ffff9d85c0fb2300 x1874531134470144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697722.003536:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134470144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697722.003540:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697722.003542:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697722.003544:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697722.003546:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697722.003554:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697722.003559:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697722/real 1787697722] req@ffff9d85c0fb3480 x1874531134470272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697722.003564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134470272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697722.003568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697722.003570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697722.003571:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470272:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697722.579069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697723.603005:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697724.626957:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697725.651088:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697726.675032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697727.122959:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697727.122970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697727.122990:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697727.123002:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5628 00000100:00080000:3.0:1787697727.123012:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697727.123019:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697727.123076:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134470400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697727.123099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697727.123102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697727.123104:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697727.123108:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5628 00000100:00080000:3.0:1787697727.123112:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697727.123116:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697727.123125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697727.123141:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134470400:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697727.123141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134470528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697727.123151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697727.123155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697727.123156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697727.123160:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5628 00000100:00080000:3.0:1787697727.123163:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697727.123167:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697727.123176:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697727.123187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134470656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697727.123196:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697727.123200:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697727.123201:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697727.123206:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5628 00000100:00080000:3.0:1787697727.123210:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697727.123213:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697727.123218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697727.123230:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134470784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697727.123234:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697727.123243:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697727.123246:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697727.123248:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697727.123253:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5628 00000100:00080000:3.0:1787697727.123256:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697727.123260:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697727.123265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697727.123275:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134470912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697727.123284:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697727.123289:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697727/real 1787697727] req@ffff9d85c0fb1c00 x1874531134470400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697727.123289:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697727.123293:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697727.123301:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5628 00000100:00080000:3.0:1787697727.123307:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697727.123312:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697727.123316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134470400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697727.123319:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697727.123334:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134471040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697727.123338:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697727.123347:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697727.123352:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697727.123354:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134470400:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697727.123359:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697727.123362:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697727.123366:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5628 00000100:00080000:3.0:1787697727.123369:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697727.123372:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697727.123375:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697727.123383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134470528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697727.123385:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134471168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697727.123425:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697727.123438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697727/real 1787697727] req@ffff9d85c0fb3800 x1874531134470528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697727.123449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134470528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697727.123457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697727.123459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697727.123463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134470528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697727.123468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134470656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697727.123491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697727.123502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697727/real 1787697727] req@ffff9d85c0fb0a80 x1874531134470656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697727.123513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134470656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697727.123520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697727.123523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697727.123526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134470656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697727.123532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134470784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697727.123547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697727.123556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697727/real 1787697727] req@ffff9d85c0fb0e00 x1874531134470784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697727.123566:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134470784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697727.123573:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697727.123576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697727.123579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134470784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697727.123584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470912:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697727.123599:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697727.123608:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697727/real 1787697727] req@ffff9d85c0fb2680 x1874531134470912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697727.123618:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134470912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697727.123625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697727.123628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697727.123631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134470912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697727.123654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697727.123676:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697727.123686:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697727/real 1787697727] req@ffff9d85c0fb2a00 x1874531134471040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697727.123697:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134471040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697727.123705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697727.123707:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697727.123710:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697727.123727:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697727.123742:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697727.123751:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697727/real 1787697727] req@ffff9d85c0fb3100 x1874531134471168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697727.123760:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134471168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697727.123767:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697727.123770:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697727.123773:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471168:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697727.699042:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697728.723017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697729.747011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697730.771020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697731.795002:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697732.242988:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697732.242997:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697732.243009:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697732.243017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5634 00000100:00080000:3.0:1787697732.243025:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697732.243030:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697732.243066:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134471296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697732.243086:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697732.243089:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697732.243090:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697732.243094:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5634 00000100:00080000:3.0:1787697732.243096:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697732.243098:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697732.243106:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697732.243114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134471424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697732.243121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697732.243123:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697732.243124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697732.243128:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134471296:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697732.243130:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5634 00000100:00080000:3.0:1787697732.243135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697732.243138:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697732.243142:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697732.243151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134471552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697732.243159:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697732.243163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697732.243164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697732.243168:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5634 00000100:00080000:3.0:1787697732.243171:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697732.243175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697732.243179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697732.243191:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134471680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697732.243198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697732.243201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787697732.243202:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697732.243203:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697732.243208:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5634 00000100:00080000:3.0:1787697732.243211:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697732.243214:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697732.243218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697732.243225:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134471808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697732.243231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697732.243233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697732.243234:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697732.243236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5634 00000100:00080000:3.0:1787697732.243238:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697732.243243:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697732.243244:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697732/real 1787697732] req@ffff9d85c0fb3b80 x1874531134471296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697732.243245:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697732.243255:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134471936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697732.243263:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697732.243264:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697732.243265:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697732.243267:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5634 00000100:00080000:3.0:1787697732.243269:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697732.243272:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697732.243274:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697732.243279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134472064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697732.243310:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134471296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697732.243356:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697732.243359:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697732.243362:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134471296:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697732.243378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134471424:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697732.243439:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697732.243466:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697732/real 1787697732] req@ffff9d85c0fb0380 x1874531134471424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697732.243475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134471424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697732.243490:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697732.243492:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697732.243494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134471424:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697732.243498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134471552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697732.243510:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697732.243532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697732/real 1787697732] req@ffff9d85c0fb2d80 x1874531134471552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697732.243610:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134471552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697732.243619:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697732.243622:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697732.243625:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134471552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697732.243640:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134471680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697732.243662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697732.243673:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697732/real 1787697732] req@ffff9d85c0fb3100 x1874531134471680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697732.243685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134471680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697732.243693:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697732.243695:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697732.243699:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134471680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697732.243704:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471808:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697732.243721:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697732.243731:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697732/real 1787697732] req@ffff9d85c0fb2a00 x1874531134471808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697732.243742:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134471808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697732.243749:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697732.243752:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697732.243755:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697732.243760:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697732.243777:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697732.243787:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697732/real 1787697732] req@ffff9d85c0fb2680 x1874531134471936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697732.243799:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134471936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697732.243806:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697732.243809:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697732.243812:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134471936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697732.243817:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697732.243834:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697732.243843:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697732/real 1787697732] req@ffff9d85c0fb0e00 x1874531134472064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697732.243879:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134472064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697732.243888:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697732.243892:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697732.243895:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472064:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697732.819096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697733.843011:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697734.867108:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697735.891004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697736.915010:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697737.362991:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697737.363000:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697737.363004:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697737.363012:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5639 00000100:00080000:2.0:1787697737.363021:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697737.363027:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697737.363064:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134472192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697737.363090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697737.363093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697737.363096:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697737.363099:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5639 00000100:00080000:2.0:1787697737.363102:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697737.363107:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697737.363116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697737.363139:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134472192:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697737.363142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134472320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697737.363163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697737.363166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697737.363167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697737.363173:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5639 00000100:00080000:2.0:1787697737.363177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697737.363181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697737.363186:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697737.363199:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134472448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697737.363210:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697737.363213:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697737.363215:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697737.363218:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5639 00000800:00000100:1.0:1787697737.363223:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697737.363227:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697737.363233:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697737.363239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697737.363252:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134472576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697737.363262:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697737.363265:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697737.363268:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697737.363272:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697737/real 1787697737] req@ffff9d85babd4e00 x1874531134472192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697737.363278:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5639 00000100:00080000:2.0:1787697737.363290:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697737.363300:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697737.363305:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134472192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697737.363312:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697737.363334:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697737.363334:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134472704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697737.363351:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697737.363363:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134472192:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697737.363366:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697737.363371:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697737.363373:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697737.363378:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5639 00000100:00080000:2.0:1787697737.363382:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697737.363386:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697737.363391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134472320:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787697737.363392:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697737.363416:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134472832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697737.363427:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697737.363434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697737.363436:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697737.363440:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697737.363446:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5639 00000100:00080000:2.0:1787697737.363450:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697737.363458:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697737/real 1787697737] req@ffff9d85babd7100 x1874531134472320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697737.363486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134472320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697737.363491:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787697737.363495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697737.363500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787697737.363500:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697737.363509:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134472320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697737.363516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134472448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787697737.363521:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134472960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697737.363536:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697737.363548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697737/real 1787697737] req@ffff9d85babd7b80 x1874531134472448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697737.363557:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134472448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697737.363562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697737.363571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697737.363573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134472448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697737.363578:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134472576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697737.363589:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697737.363595:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697737/real 1787697737] req@ffff9d85babd4a80 x1874531134472576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697737.363601:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134472576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697737.363606:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697737.363608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697737.363609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134472576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697737.363612:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472704:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787697737.363622:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697737.363627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697737/real 1787697737] req@ffff9d85babd4700 x1874531134472704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697737.363633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134472704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697737.363638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697737.363639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697737.363641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697737.363654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697737.363670:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697737.363680:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697737/real 1787697737] req@ffff9d85babd5f80 x1874531134472832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697737.363692:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134472832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697737.363699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697737.363702:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697737.363706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697737.363712:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787697737.363729:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697737.363737:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697737/real 1787697737] req@ffff9d85babd5180 x1874531134472960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697737.363744:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134472960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697737.363750:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697737.363752:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697737.363753:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134472960:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697737.939054:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697738.962986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697739.987041:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697741.011090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697742.035096:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697742.482953:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697742.482960:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697742.482963:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697742.482969:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5644 00000100:00080000:2.0:1787697742.482978:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697742.482984:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697742.483019:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134473088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697742.483042:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697742.483045:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697742.483046:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697742.483050:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5644 00000100:00080000:2.0:1787697742.483053:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697742.483056:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697742.483064:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697742.483075:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134473216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697742.483083:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697742.483087:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697742.483088:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697742.483091:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5644 00000100:00080000:2.0:1787697742.483095:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697742.483101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134473088:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697742.483102:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697742.483113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697742.483127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134473344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697742.483136:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697742.483139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697742.483141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697742.483145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5644 00000100:00080000:2.0:1787697742.483149:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697742.483153:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697742.483158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697742.483170:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134473472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697742.483179:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697742.483183:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697742.483184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697742.483187:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5644 00000100:00080000:2.0:1787697742.483192:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697742.483201:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697742.483203:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787697742.483212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697742.483222:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134473600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697742.483228:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697742.483230:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697742.483231:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697742.483233:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5644 00000100:00080000:2.0:1787697742.483235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697742.483237:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697742.483240:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697742.483244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134473728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697742.483249:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697742.483251:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697742.483252:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697742.483254:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5644 00000100:00080000:2.0:1787697742.483255:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697742.483257:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697742.483259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697742.483280:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134473856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697742.483344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697742/real 1787697742] req@ffff9d85babd7480 x1874531134473088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697742.483363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134473088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697742.483372:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697742.483376:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697742.483390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134473088:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697742.483415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134473216:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697742.483470:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697742.483489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697742/real 1787697742] req@ffff9d85babd5c00 x1874531134473216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697742.483500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134473216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697742.483508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697742.483511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697742.483514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134473216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697742.483520:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134473344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697742.483537:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697742.483544:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697742/real 1787697742] req@ffff9d85babd4000 x1874531134473344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697742.483555:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134473344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697742.483559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697742.483561:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697742.483563:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134473344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697742.483566:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134473472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697742.483575:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697742.483580:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697742/real 1787697742] req@ffff9d85babd4380 x1874531134473472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697742.483586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134473472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697742.483591:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697742.483592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697742.483594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134473472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697742.483596:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134473600:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697742.483605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697742.483609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697742/real 1787697742] req@ffff9d85babd6300 x1874531134473600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697742.483615:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134473600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697742.483623:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697742.483624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697742.483626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134473600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697742.483629:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134473728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697742.483636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697742.483641:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697742/real 1787697742] req@ffff9d85babd6a00 x1874531134473728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697742.483646:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134473728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697742.483650:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697742.483652:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697742.483654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134473728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697742.483656:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134473856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697742.483664:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697742.483668:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697742/real 1787697742] req@ffff9d85babd6680 x1874531134473856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697742.483673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134473856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697742.483678:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697742.483679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697742.483682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134473856:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697743.059041:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697744.083057:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697745.107121:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697746.131054:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697747.155061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697747.602958:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697747.602967:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697747.602971:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697747.602981:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5649 00000100:00080000:2.0:1787697747.602991:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697747.602997:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697747.603032:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134473984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697747.603058:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697747.603061:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697747.603063:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697747.603067:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5649 00000100:00080000:2.0:1787697747.603070:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697747.603074:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697747.603082:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697747.603094:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134474112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697747.603105:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697747.603108:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697747.603110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697747.603114:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5649 00000100:00080000:2.0:1787697747.603118:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697747.603122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697747.603128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697747.603141:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134474240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697747.603152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697747.603153:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134473984:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697747.603157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697747.603164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697747.603170:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5649 00000100:00080000:2.0:1787697747.603174:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697747.603178:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697747.603182:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697747.603197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134474368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697747.603207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697747.603217:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697747.603219:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697747.603223:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5649 00000100:00080000:2.0:1787697747.603228:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697747.603232:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697747.603237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697747.603244:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697747.603250:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134474496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697747.603263:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697747.603274:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697747.603277:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697747.603281:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5649 00000100:00080000:2.0:1787697747.603286:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697747.603303:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697747.603309:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697747.603318:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697747/real 1787697747] req@ffff9d85babd7800 x1874531134473984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697747.603325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134474624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697747.603353:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697747.603358:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697747.603360:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697747.603363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134473984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697747.603368:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5649 00000100:00080000:2.0:1787697747.603379:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697747.603387:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697747.603391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787697747.603394:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697747.603401:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697747.603407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134473984:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787697747.603413:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134474752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697747.603422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134474112:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697747.603462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697747.603474:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697747/real 1787697747] req@ffff9d85babd5500 x1874531134474112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697747.603487:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134474112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697747.603496:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697747.603500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697747.603503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134474112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697747.603536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134474240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697747.603556:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697747.603567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697747/real 1787697747] req@ffff9d85babd6d80 x1874531134474240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697747.603576:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134474240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697747.603590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697747.603593:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697747.603597:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134474240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697747.603603:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134474368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697747.603620:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697747.603630:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697747/real 1787697747] req@ffff9d85babd6680 x1874531134474368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697747.603639:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134474368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697747.603647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697747.603651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697747.603654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134474368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697747.603660:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134474496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697747.603677:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697747.603686:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697747/real 1787697747] req@ffff9d85babd6a00 x1874531134474496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697747.603694:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134474496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697747.603702:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697747.603705:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697747.603709:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134474496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697747.603715:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134474624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697747.603739:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697747.603748:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697747/real 1787697747] req@ffff9d85babd6300 x1874531134474624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697747.603758:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134474624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697747.603767:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697747.603770:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697747.603773:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134474624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697747.603780:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134474752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697747.603796:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697747.603804:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697747/real 1787697747] req@ffff9d85babd4380 x1874531134474752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697747.603814:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134474752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697747.603823:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697747.603826:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697747.603830:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134474752:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697748.179152:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697749.203020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697750.227056:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697751.251004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697752.275065:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697752.722955:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697752.722963:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697752.722966:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697752.722973:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5654 00000100:00080000:2.0:1787697752.722982:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697752.722987:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697752.723018:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134474880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697752.723040:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697752.723042:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697752.723044:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697752.723047:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5654 00000100:00080000:2.0:1787697752.723050:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697752.723054:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697752.723063:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697752.723077:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134475008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697752.723085:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134474880:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697752.723090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697752.723094:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697752.723095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697752.723098:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5654 00000100:00080000:2.0:1787697752.723101:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697752.723104:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697752.723108:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697752.723117:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134475136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697752.723123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697752.723125:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697752.723126:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697752.723130:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5654 00000100:00080000:2.0:1787697752.723132:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697752.723134:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697752.723137:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697752.723139:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697752.723149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134475264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697752.723158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697752.723161:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697752.723163:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697752.723167:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5654 00000100:00080000:2.0:1787697752.723171:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697752.723174:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697752/real 1787697752] req@ffff9d85babd4000 x1874531134474880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697752.723179:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697752.723188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697752.723193:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134474880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697752.723199:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134475392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697752.723217:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697752.723222:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697752.723228:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134474880:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697752.723235:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697752.723238:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697752.723239:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697752.723243:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5654 00000100:00080000:2.0:1787697752.723246:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697752.723249:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697752.723252:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475008:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787697752.723254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697752.723265:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134475520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697752.723273:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697752.723276:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697752.723278:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697752.723281:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697752.723285:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5654 00000100:00080000:2.0:1787697752.723288:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697752.723290:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697752/real 1787697752] req@ffff9d85babd5c00 x1874531134475008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697752.723292:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697752.723299:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697752.723305:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134475008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697752.723312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134475648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697752.723313:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697752.723325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697752.723329:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697752.723333:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697752.723345:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697752.723352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697752/real 1787697752] req@ffff9d85babd7480 x1874531134475136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697752.723359:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134475136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697752.723364:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697752.723365:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697752.723367:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697752.723370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697752.723378:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697752.723383:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697752/real 1787697752] req@ffff9d85babd5180 x1874531134475264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697752.723389:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134475264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697752.723394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697752.723395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697752.723397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697752.723399:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134475392:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697752.723407:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697752.723412:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697752/real 1787697752] req@ffff9d85babd5f80 x1874531134475392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697752.723417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134475392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697752.723422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697752.723424:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697752.723425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134475392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697752.723434:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134475520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697752.723443:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697752.723448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697752/real 1787697752] req@ffff9d85babd4700 x1874531134475520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697752.723454:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134475520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697752.723458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697752.723460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697752.723462:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134475520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697752.723464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134475648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697752.723472:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697752.723476:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697752/real 1787697752] req@ffff9d85babd4a80 x1874531134475648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697752.723482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134475648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697752.723487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697752.723488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697752.723490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134475648:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697753.299019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697754.323061:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697755.347045:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697756.371066:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697757.395046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697757.842998:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697757.843009:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697757.843014:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697757.843023:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5659 00000100:00080000:1.0:1787697757.843035:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697757.843042:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697757.843084:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134475776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697757.843106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697757.843109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697757.843110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697757.843125:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5659 00000100:00080000:1.0:1787697757.843129:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697757.843133:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697757.843142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697757.843153:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134475904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697757.843160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697757.843162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697757.843164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697757.843166:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5659 00000100:00080000:1.0:1787697757.843169:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697757.843171:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697757.843174:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697757.843180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134476032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697757.843187:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134475776:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697757.843189:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697757.843194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697757.843197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697757.843200:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5659 00000100:00080000:1.0:1787697757.843203:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697757.843205:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697757.843208:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697757.843216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134476160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697757.843222:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697757.843224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697757.843225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697757.843227:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5659 00000100:00080000:1.0:1787697757.843229:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697757.843232:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697757.843234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697757.843240:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134476288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697757.843245:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697757.843248:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697757.843249:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697757.843252:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5659 00000100:00080000:1.0:1787697757.843261:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697757.843263:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697757.843266:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697757.843273:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697757.843274:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134476416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697757.843286:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697757.843288:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697757.843294:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697757.843296:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5659 00000100:00080000:1.0:1787697757.843300:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697757.843302:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697757.843305:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697757.843314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134476544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697757.843331:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697757/real 1787697757] req@ffff9d85b9d2c700 x1874531134475776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697757.843352:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134475776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697757.843363:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697757.843367:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697757.843371:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134475776:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697757.843379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475904:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697757.843402:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697757.843414:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697757/real 1787697757] req@ffff9d85b9d2c380 x1874531134475904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697757.843427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134475904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697757.843436:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697757.843439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697757.843443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134475904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697757.843468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697757.843489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697757.843509:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697757/real 1787697757] req@ffff9d85b9d2ed80 x1874531134476032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697757.843521:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134476032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697757.843530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697757.843533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697757.843536:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697757.843542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697757.843559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697757.843568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697757/real 1787697757] req@ffff9d85b9d2d500 x1874531134476160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697757.843580:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134476160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697757.843589:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697757.843592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697757.843595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697757.843602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134476288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697757.843619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697757.843629:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697757/real 1787697757] req@ffff9d85b9d2e680 x1874531134476288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697757.843640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134476288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697757.843649:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697757.843652:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697757.843655:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134476288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697757.843661:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134476416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697757.843676:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697757.843685:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697757/real 1787697757] req@ffff9d85b9d2dc00 x1874531134476416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697757.843696:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134476416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697757.843703:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697757.843707:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697757.843710:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134476416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697757.843715:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134476544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697757.843734:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697757.843744:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697757/real 1787697757] req@ffff9d85b9d2f800 x1874531134476544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697757.843755:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134476544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697757.843763:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697757.843766:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697757.843769:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134476544:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697758.419074:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697759.443033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697760.466975:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697761.491018:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697762.514956:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697762.962939:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697762.962946:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697762.962949:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697762.962957:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5664 00000100:00080000:1.0:1787697762.962965:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697762.962971:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697762.963001:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134476672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697762.963025:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697762.963047:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697762.963048:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697762.963051:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5664 00000100:00080000:1.0:1787697762.963053:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697762.963056:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697762.963064:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697762.963072:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134476800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697762.963079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697762.963081:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697762.963082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697762.963085:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5664 00000100:00080000:1.0:1787697762.963088:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697762.963090:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697762.963093:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697762.963099:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134476928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697762.963106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697762.963110:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697762.963111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697762.963115:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134476672:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697762.963117:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5664 00000100:00080000:1.0:1787697762.963123:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697762.963129:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697762.963133:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697762.963146:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134477056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697762.963156:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697762.963158:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697762.963160:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697762.963162:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5664 00000100:00080000:1.0:1787697762.963166:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697762.963169:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697762.963173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697762.963181:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134477184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697762.963185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697762.963191:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697762.963193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697762.963194:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697762.963197:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5664 00000100:00080000:1.0:1787697762.963199:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697762.963202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697762.963205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697762.963211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134477312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697762.963216:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697762.963218:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697762.963219:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697762.963220:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5664 00000100:00080000:1.0:1787697762.963223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697762.963225:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697762.963228:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697762.963235:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697762/real 1787697762] req@ffff9d85b9d2f800 x1874531134476672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697762.963237:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134477440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697762.963271:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134476672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697762.963282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697762.963287:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697762.963290:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134476672:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697762.963297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476800:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697762.963317:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697762.963326:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697762/real 1787697762] req@ffff9d85b9d2dc00 x1874531134476800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697762.963334:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134476800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697762.963339:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697762.963341:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697762.963343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697762.963361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697762.963371:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697762.963377:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697762/real 1787697762] req@ffff9d85b9d2e680 x1874531134476928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697762.963384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134476928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697762.963389:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697762.963391:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697762.963392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134476928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697762.963396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697762.963413:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697762.963419:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697762/real 1787697762] req@ffff9d85b9d2d500 x1874531134477056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697762.963426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134477056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697762.963431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697762.963432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697762.963434:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697762.963438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134477184:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697762.963447:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697762.963451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697762/real 1787697762] req@ffff9d85b9d2ed80 x1874531134477184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697762.963457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134477184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697762.963462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697762.963464:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697762.963471:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134477184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697762.963475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134477312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697762.963485:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697762.963490:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697762/real 1787697762] req@ffff9d85b9d2c380 x1874531134477312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697762.963496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134477312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697762.963501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697762.963503:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697762.963506:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134477312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697762.963510:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134477440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697762.963527:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697762.963536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697762/real 1787697762] req@ffff9d85b9d2c700 x1874531134477440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697762.963548:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134477440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697762.963554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697762.963556:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697762.963558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134477440:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697763.539012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697764.562996:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697765.587014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697766.611014:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697767.634999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697768.082982:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697768.082989:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697768.082992:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697768.082999:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5669 00000100:00080000:1.0:1787697768.083008:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697768.083013:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697768.083040:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134477568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697768.083063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697768.083065:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697768.083067:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697768.083069:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5669 00000100:00080000:1.0:1787697768.083071:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697768.083073:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697768.083079:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697768.083086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134477696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697768.083093:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697768.083095:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697768.083096:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697768.083098:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5669 00000100:00080000:1.0:1787697768.083100:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697768.083103:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697768.083107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697768.083109:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134477568:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697768.083115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134477824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697768.083121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697768.083123:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697768.083125:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697768.083127:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5669 00000100:00080000:1.0:1787697768.083129:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697768.083131:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697768.083133:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697768.083139:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134477952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697768.083144:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697768.083146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697768.083148:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697768.083152:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5669 00000100:00080000:1.0:1787697768.083155:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697768.083160:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697768.083163:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697768.083167:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697768.083180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134478080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697768.083189:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697768.083193:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697768.083195:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697768.083199:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5669 00000100:00100000:0.0:1787697768.083203:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697768/real 1787697768] req@ffff9d85b9d2fb80 x1874531134477568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697768.083205:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697768.083212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697768.083218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697768.083221:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134477568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697768.083228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134478208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697768.083231:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697768.083236:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697768.083242:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134477568:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697768.083250:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697768.083252:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477696:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697768.083254:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697768.083258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697768.083262:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5669 00000100:00080000:1.0:1787697768.083264:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697768.083267:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697768.083270:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697768.083274:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697768.083283:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134478336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697768.083286:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697768/real 1787697768] req@ffff9d85b9d2e300 x1874531134477696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697768.083305:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134477696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697768.083310:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697768.083312:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697768.083313:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697768.083329:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697768.083341:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697768.083347:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697768/real 1787697768] req@ffff9d85b9d2ce00 x1874531134477824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697768.083354:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134477824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697768.083358:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697768.083360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697768.083361:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697768.083364:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697768.083373:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697768.083378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697768/real 1787697768] req@ffff9d85b9d2c000 x1874531134477952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697768.083384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134477952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697768.083388:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697768.083390:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697768.083391:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134477952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697768.083394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478080:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697768.083402:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697768.083407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697768/real 1787697768] req@ffff9d85b9d2ca80 x1874531134478080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697768.083413:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134478080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697768.083417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697768.083419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697768.083421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697768.083428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697768.083438:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697768.083443:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697768/real 1787697768] req@ffff9d85b9d2f480 x1874531134478208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697768.083449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134478208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697768.083453:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697768.083455:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697768.083456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697768.083459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697768.083467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697768.083472:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697768/real 1787697768] req@ffff9d85b9d2c700 x1874531134478336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697768.083478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134478336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697768.083482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697768.083484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697768.083486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478336:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697768.659042:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697769.683015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697770.706996:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697771.730985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697772.754986:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697773.202944:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697773.202952:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697773.202956:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697773.202966:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5674 00000100:00080000:2.0:1787697773.202977:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697773.202983:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697773.203013:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134478464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697773.203037:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697773.203040:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697773.203042:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697773.203046:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5674 00000100:00080000:2.0:1787697773.203050:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697773.203053:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697773.203061:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697773.203073:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134478592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697773.203084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134478464:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697773.203086:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697773.203093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697773.203095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697773.203099:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5674 00000100:00080000:2.0:1787697773.203102:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697773.203105:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697773.203109:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697773.203119:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134478720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697773.203127:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697773.203130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697773.203132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697773.203136:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5674 00000100:00080000:2.0:1787697773.203139:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697773.203142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697773.203145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697773.203148:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697773.203157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134478848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697773.203164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697773.203166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697773.203167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697773.203169:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5674 00000100:00080000:2.0:1787697773.203171:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697773.203173:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697773.203176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697773.203182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134478976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697773.203188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697773.203191:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697773.203194:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697773/real 1787697773] req@ffff9d85babd7b80 x1874531134478464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697773.203194:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697773.203199:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5674 00000100:00080000:2.0:1787697773.203202:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697773.203207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697773.203213:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697773.203221:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134479104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697773.203227:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134478464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697773.203237:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697773.203240:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697773.203242:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697773.203242:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697773.203250:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5674 00000100:00080000:0.0:1787697773.203252:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697773.203254:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697773.203259:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697773.203262:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134478464:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787697773.203264:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697773.203275:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134479232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697773.203290:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134478592:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697773.203314:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697773.203325:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697773/real 1787697773] req@ffff9d85babd7100 x1874531134478592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697773.203335:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134478592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697773.203342:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697773.203345:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697773.203348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134478592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697773.203353:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134478720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697773.203370:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697773.203380:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697773/real 1787697773] req@ffff9d85babd4e00 x1874531134478720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697773.203391:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134478720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697773.203406:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697773.203409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697773.203412:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134478720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697773.203418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134478848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697773.203438:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697773.203448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697773/real 1787697773] req@ffff9d85babd4a80 x1874531134478848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697773.203458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134478848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697773.203464:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697773.203467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697773.203470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134478848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697773.203474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478976:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697773.203492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697773.203501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697773/real 1787697773] req@ffff9d85babd4700 x1874531134478976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697773.203513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134478976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697773.203520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697773.203523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697773.203526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134478976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697773.203531:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134479104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697773.203565:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697773.203577:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697773/real 1787697773] req@ffff9d85babd5f80 x1874531134479104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697773.203588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134479104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697773.203595:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697773.203598:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697773.203601:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134479104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697773.203607:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134479232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697773.203624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697773.203634:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697773/real 1787697773] req@ffff9d85babd5180 x1874531134479232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697773.203644:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134479232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697773.203651:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697773.203654:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697773.203657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134479232:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697773.779005:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697774.802980:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697775.827008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697776.851038:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697777.875019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697778.322992:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697778.323002:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697778.323006:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697778.323016:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5680 00000100:00080000:1.0:1787697778.323028:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697778.323034:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697778.323074:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134479360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697778.323103:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697778.323108:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697778.323110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697778.323115:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5680 00000100:00080000:1.0:1787697778.323119:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697778.323123:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697778.323136:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697778.323146:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134479360:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697778.323152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134479488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697778.323164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697778.323169:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697778.323171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697778.323174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5680 00000100:00080000:1.0:1787697778.323178:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697778.323183:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697778.323188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697778.323203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134479616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697778.323212:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697778.323216:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697778.323218:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697778.323222:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5680 00000100:00080000:1.0:1787697778.323226:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697778.323229:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697778.323234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697778.323244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134479744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697778.323251:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697778.323254:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697778.323255:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697778.323257:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5680 00000100:00080000:1.0:1787697778.323259:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697778.323262:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697778.323265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697778.323271:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134479488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697778.323276:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134479872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697778.323285:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697778.323288:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697778.323289:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697778.323292:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5680 00000100:00080000:1.0:1787697778.323303:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697778.323306:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697778.323310:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697778.323314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134479616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787697778.323322:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134480000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697778.323334:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697778.323336:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697778.323338:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697778.323341:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5680 00000100:00100000:0.0:1787697778.323344:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134479744:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787697778.323345:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697778.323352:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697778.323355:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697778.323364:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134480128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697778.323377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134479872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697778.323396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697778.323409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480128:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697778.899091:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697779.922994:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697780.947060:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697781.395127:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697781.395169:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697781.395184:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697781.395222:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697781.395230:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697781.395238:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697781.395246:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697781.395254:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697781.395264:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787697781.395273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697778/real 1787697781] req@ffff9d85b9d2dc00 x1874531134479872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697781.395292:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134479872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697781.395300:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697781.395304:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697781.395308:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134479872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697781.395315:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697778/real 1787697781] req@ffff9d85b9d2f800 x1874531134480000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697781.395333:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134480000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697781.395345:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697781.395349:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697781.395352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697781.395358:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697778/real 1787697781] req@ffff9d85b9d2c700 x1874531134480128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697781.395368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134480128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697781.395374:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697781.395377:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697781.395379:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697781.395384:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697778/real 1787697781] req@ffff9d85b9d2ed80 x1874531134479488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697781.395393:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134479488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697781.395400:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697781.395409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697781.395412:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134479488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697781.395418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697778/real 1787697781] req@ffff9d85b9d2d500 x1874531134479616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697781.395428:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134479616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697781.395436:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697781.395439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697781.395442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134479616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697781.395447:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697778/real 1787697781] req@ffff9d85b9d2e680 x1874531134479744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697781.395458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134479744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697781.395465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697781.395468:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697781.395471:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134479744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697781.395477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697778/real 1787697781] req@ffff9d85b9d2c380 x1874531134479360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697781.395489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134479360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697781.395496:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697781.395499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697781.395502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134479360:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787697781.970951:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697782.994990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697783.442941:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697783.442949:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697783.442952:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697783.442960:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5685 00000100:00080000:1.0:1787697783.442968:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697783.442974:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697783.443018:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134480256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697783.443048:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697783.443051:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697783.443053:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697783.443057:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5685 00000100:00080000:1.0:1787697783.443060:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697783.443064:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697783.443073:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697783.443087:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134480384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697783.443097:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134480256:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697783.443099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697783.443107:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697783.443109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697783.443113:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5685 00000100:00080000:1.0:1787697783.443116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697783.443120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697783.443126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697783.443138:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134480512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697783.443147:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697783.443150:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697783.443152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697783.443155:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5685 00000100:00080000:1.0:1787697783.443158:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697783.443160:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697783.443166:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697783.443173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697783.443184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134480640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697783.443193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697783.443196:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697783.443201:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697783/real 1787697783] req@ffff9d85b9d2f480 x1874531134480256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697783.443201:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697783.443208:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5685 00000100:00080000:1.0:1787697783.443216:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697783.443224:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697783.443232:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697783.443240:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134480256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697783.443246:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134480768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697783.443256:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697783.443261:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697783.443264:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697783.443267:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697783.443272:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697783.443273:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134480256:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697783.443281:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5685 00000100:00080000:1.0:1787697783.443286:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697783.443290:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697783.443294:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697783.443304:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134480384:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697783.443307:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134480896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697783.443318:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697783.443321:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697783.443323:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697783.443327:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5685 00000100:00080000:1.0:1787697783.443331:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697783.443335:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697783.443337:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697783.443343:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697783.443353:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697783/real 1787697783] req@ffff9d85b9d2ca80 x1874531134480384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697783.443358:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134481024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697783.443380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134480384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697783.443390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697783.443393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697783.443398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134480384:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697783.443405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134480512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697783.443426:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697783.443436:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697783/real 1787697783] req@ffff9d85b9d2c000 x1874531134480512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697783.443448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134480512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697783.443460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697783.443464:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697783.443467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134480512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697783.443473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134480640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697783.443499:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697783.443509:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697783/real 1787697783] req@ffff9d85b9d2ce00 x1874531134480640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697783.443520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134480640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697783.443527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697783.443530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697783.443534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134480640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697783.443539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480768:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697783.443559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697783.443570:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697783/real 1787697783] req@ffff9d85b9d2e300 x1874531134480768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697783.443581:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134480768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697783.443588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697783.443591:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697783.443594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697783.443613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697783.443631:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697783.443641:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697783/real 1787697783] req@ffff9d85b9d2fb80 x1874531134480896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697783.443651:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134480896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697783.443658:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697783.443661:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697783.443664:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134480896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697783.443670:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697783.443699:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697783.443709:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697783/real 1787697783] req@ffff9d85b9d2f480 x1874531134481024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697783.443720:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134481024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697783.443727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697783.443730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697783.443733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481024:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697784.019003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697785.043006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697786.067001:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697787.091006:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697788.115007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697788.562984:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697788.562998:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697788.563003:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697788.563017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5690 00000100:00080000:3.0:1787697788.563030:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697788.563039:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697788.563086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134481152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697788.563118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697788.563122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697788.563124:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697788.563129:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5690 00000100:00080000:3.0:1787697788.563133:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697788.563138:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697788.563149:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697788.563163:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134481152:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697788.563166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134481280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697788.563178:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697788.563181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697788.563183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697788.563187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5690 00000100:00080000:3.0:1787697788.563191:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697788.563195:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697788.563202:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697788.563214:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134481408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697788.563224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697788.563228:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697788.563230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697788.563232:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697788.563236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5690 00000100:00080000:3.0:1787697788.563250:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697788.563258:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697788.563263:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697788.563276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134481536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697788.563282:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697788.563285:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697788.563286:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697788.563289:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5690 00000100:00080000:3.0:1787697788.563293:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697788.563294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697788/real 1787697788] req@ffff9d85c0fb0a80 x1874531134481152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697788.563298:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697788.563304:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697788.563315:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134481664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697788.563322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134481152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697788.563323:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697788.563330:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697788.563332:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697788.563335:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5690 00000100:00080000:3.0:1787697788.563337:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697788.563343:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697788.563348:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697788.563349:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697788.563354:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697788.563358:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134481152:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697788.563365:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134481792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697788.563375:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697788.563378:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697788.563380:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697788.563385:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134481280:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697788.563388:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5690 00000100:00080000:3.0:1787697788.563394:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697788.563398:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697788.563402:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697788.563415:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134481920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697788.563419:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697788.563435:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697788/real 1787697788] req@ffff9d85c0fb3800 x1874531134481280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697788.563445:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134481280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697788.563451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697788.563454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697788.563456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134481280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697788.563460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134481408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697788.563474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697788.563481:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697788/real 1787697788] req@ffff9d85c0fb1c00 x1874531134481408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697788.563492:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134481408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697788.563505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697788.563508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697788.563512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134481408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697788.563517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134481536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697788.563532:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697788.563539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697788/real 1787697788] req@ffff9d85c0fb3480 x1874531134481536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697788.563546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134481536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697788.563551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697788.563553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697788.563555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134481536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697788.563557:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481664:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697788.563567:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697788.563572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697788/real 1787697788] req@ffff9d85c0fb2300 x1874531134481664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697788.563578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134481664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697788.563582:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697788.563584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697788.563586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697788.563605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697788.563619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697788.563625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697788/real 1787697788] req@ffff9d85c0fb1880 x1874531134481792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697788.563632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134481792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697788.563637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697788.563639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697788.563641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697788.563653:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697788.563662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697788.563667:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697788/real 1787697788] req@ffff9d85c0fb1500 x1874531134481920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697788.563674:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134481920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697788.563678:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697788.563680:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697788.563681:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134481920:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697789.139043:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697790.163005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697791.187004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697792.210977:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697793.235016:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697793.682970:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697793.682980:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697793.682984:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697793.682994:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5695 00000100:00080000:2.0:1787697793.683006:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697793.683014:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697793.683063:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134482048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697793.683092:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697793.683094:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697793.683096:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697793.683099:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5695 00000100:00080000:2.0:1787697793.683102:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697793.683105:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697793.683114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697793.683126:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134482176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697793.683134:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697793.683135:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134482048:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697793.683138:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697793.683141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697793.683145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5695 00000100:00080000:2.0:1787697793.683147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697793.683150:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697793.683153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697793.683161:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134482304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697793.683168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697793.683171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697793.683172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697793.683174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5695 00000100:00080000:2.0:1787697793.683176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697793.683177:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697793.683180:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697793.683187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134482432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697793.683192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697793.683195:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697793.683196:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697793.683197:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5695 00000100:00080000:2.0:1787697793.683200:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697793.683203:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697793.683205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697793.683211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134482560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697793.683218:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697793.683219:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697793.683223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697793.683225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697793.683228:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5695 00000100:00080000:2.0:1787697793.683231:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697793.683234:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697793.683237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697793.683243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134482688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697793.683248:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697793.683250:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697793.683251:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697793.683253:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5695 00000100:00080000:2.0:1787697793.683256:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697793.683258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697793.683261:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697793.683272:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134482816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697793.683273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697793/real 1787697793] req@ffff9d85babd7480 x1874531134482048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697793.683301:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134482048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697793.683309:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697793.683312:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697793.683315:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134482048:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697793.683335:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134482176:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697793.683353:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697793.683361:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697793/real 1787697793] req@ffff9d85babd5c00 x1874531134482176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697793.683368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134482176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697793.683373:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697793.683375:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697793.683377:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134482176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697793.683380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134482304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697793.683391:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697793.683396:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697793/real 1787697793] req@ffff9d85babd4000 x1874531134482304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697793.683411:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134482304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697793.683422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697793.683424:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697793.683427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134482304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697793.683430:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134482432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697793.683440:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697793.683446:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697793/real 1787697793] req@ffff9d85babd4380 x1874531134482432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697793.683453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134482432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697793.683457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697793.683459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697793.683460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134482432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697793.683463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134482560:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697793.683475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697793.683485:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697793/real 1787697793] req@ffff9d85babd6300 x1874531134482560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697793.683496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134482560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697793.683534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697793.683538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697793.683542:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134482560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697793.683547:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134482688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697793.683567:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697793.683575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697793/real 1787697793] req@ffff9d85babd6a00 x1874531134482688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697793.683582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134482688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697793.683587:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697793.683589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697793.683591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134482688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697793.683595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134482816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697793.683606:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697793.683615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697793/real 1787697793] req@ffff9d85babd6680 x1874531134482816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697793.683626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134482816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697793.683634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697793.683637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697793.683640:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134482816:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697794.259026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697795.283017:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697796.307074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697797.331079:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697798.355000:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697798.802959:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697798.802968:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697798.802973:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697798.802985:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5700 00000100:00080000:2.0:1787697798.802996:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697798.803002:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697798.803044:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134482944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697798.803079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697798.803082:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697798.803083:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697798.803086:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5700 00000100:00080000:2.0:1787697798.803088:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697798.803090:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697798.803099:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697798.803109:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134483072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697798.803116:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697798.803118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697798.803119:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697798.803122:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5700 00000100:00080000:2.0:1787697798.803123:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697798.803125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697798.803128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697798.803134:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134483200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697798.803139:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134482944:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697798.803142:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697798.803147:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697798.803149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697798.803152:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5700 00000100:00080000:2.0:1787697798.803154:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697798.803156:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697798.803158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697798.803167:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134483328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697798.803180:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697798.803182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697798.803183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697798.803186:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5700 00000100:00080000:2.0:1787697798.803189:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697798.803191:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697798.803194:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697798.803202:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134483456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697798.803207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697798.803212:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697798.803212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697798.803214:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697798.803219:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5700 00000100:00080000:2.0:1787697798.803221:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697798.803224:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697798.803226:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697798.803234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134483584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697798.803244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697798.803246:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697798.803247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697798.803250:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5700 00000100:00080000:2.0:1787697798.803252:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697798.803255:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697798.803258:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697798.803260:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697798/real 1787697798] req@ffff9d85babd6d80 x1874531134482944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697798.803272:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134483712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697798.803295:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134482944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697798.803308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697798.803312:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697798.803316:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134482944:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697798.803325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483072:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697798.803350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697798.803361:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697798/real 1787697798] req@ffff9d85babd5500 x1874531134483072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697798.803369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134483072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697798.803375:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697798.803377:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697798.803380:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697798.803407:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697798.803440:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697798.803453:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697798/real 1787697798] req@ffff9d85babd7800 x1874531134483200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697798.803465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134483200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697798.803479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697798.803483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697798.803487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697798.803493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697798.803510:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697798.803518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697798/real 1787697798] req@ffff9d85babd6680 x1874531134483328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697798.803525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134483328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697798.803530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697798.803532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697798.803534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697798.803537:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134483456:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697798.803548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697798.803553:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697798/real 1787697798] req@ffff9d85babd6a00 x1874531134483456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697798.803560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134483456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697798.803564:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697798.803566:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697798.803568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134483456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697798.803572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134483584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697798.803581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697798.803585:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697798/real 1787697798] req@ffff9d85babd6300 x1874531134483584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697798.803591:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134483584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697798.803596:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697798.803597:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697798.803599:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134483584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697798.803602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134483712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697798.803610:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697798.803615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697798/real 1787697798] req@ffff9d85babd4380 x1874531134483712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697798.803620:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134483712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697798.803625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697798.803627:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697798.803628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134483712:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697799.379015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697800.403020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697801.426948:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697802.451059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697803.474987:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697803.922958:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697803.922968:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697803.922972:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697803.922983:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5705 00000100:00080000:3.0:1787697803.922993:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697803.923000:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697803.923034:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134483840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697803.923059:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697803.923063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697803.923065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697803.923069:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5705 00000100:00080000:3.0:1787697803.923072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697803.923076:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697803.923086:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697803.923099:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134483840:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697803.923100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134483968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697803.923113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697803.923116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697803.923118:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697803.923121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5705 00000100:00080000:3.0:1787697803.923124:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697803.923128:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697803.923132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697803.923145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134484096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697803.923153:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697803.923157:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697803.923159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697803.923163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5705 00000100:00080000:3.0:1787697803.923166:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697803.923170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697803.923172:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697803.923179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697803.923191:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134484224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697803.923200:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697803.923203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697803.923207:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697803/real 1787697803] req@ffff9d85c0fb1f80 x1874531134483840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697803.923207:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697803.923216:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5705 00000100:00080000:3.0:1787697803.923222:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697803.923227:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134483840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697803.923228:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697803.923236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697803.923239:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697803.923244:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697803.923247:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134483840:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697803.923250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134484352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697803.923261:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697803.923265:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697803.923266:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697803.923267:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483968:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697803.923275:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5705 00000100:00080000:3.0:1787697803.923280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697803.923284:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697803.923288:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697803.923289:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697803.923298:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697803/real 1787697803] req@ffff9d85c0fb0000 x1874531134483968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697803.923304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134484480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697803.923309:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134483968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697803.923342:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697803.923346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697803.923350:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134483968:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697803.923353:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697803.923357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484096:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787697803.923359:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697803.923363:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697803.923368:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5705 00000100:00080000:3.0:1787697803.923372:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697803.923377:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697803.923382:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697803.923386:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697803.923398:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134484608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697803.923400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697803/real 1787697803] req@ffff9d85c0fb0700 x1874531134484096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697803.923421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134484096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697803.923439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697803.923443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697803.923446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697803.923451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697803.923465:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697803.923471:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697803/real 1787697803] req@ffff9d85c0fb1500 x1874531134484224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697803.923478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134484224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697803.923483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697803.923485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697803.923487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697803.923490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134484352:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697803.923498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697803.923503:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697803/real 1787697803] req@ffff9d85c0fb1880 x1874531134484352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697803.923509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134484352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697803.923513:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697803.923515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697803.923517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134484352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697803.923527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134484480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697803.923537:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697803.923542:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697803/real 1787697803] req@ffff9d85c0fb2300 x1874531134484480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697803.923548:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134484480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697803.923552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697803.923559:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697803.923561:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134484480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697803.923564:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134484608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697803.923573:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697803.923578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697803/real 1787697803] req@ffff9d85c0fb3480 x1874531134484608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697803.923584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134484608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697803.923588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697803.923590:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697803.923591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134484608:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697804.499011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697805.522999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697806.547025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697807.570975:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697808.595020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697809.042953:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697809.042964:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697809.042968:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697809.042977:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5710 00000100:00080000:3.0:1787697809.042987:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697809.042995:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697809.043034:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134484736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697809.043063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697809.043067:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697809.043068:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697809.043072:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5710 00000100:00080000:3.0:1787697809.043076:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697809.043079:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697809.043087:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697809.043101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134484864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697809.043108:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134484736:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697809.043114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697809.043119:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697809.043120:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697809.043124:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5710 00000100:00080000:3.0:1787697809.043128:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697809.043133:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697809.043138:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697809.043150:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134484992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697809.043159:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697809.043162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697809.043164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697809.043170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697809.043173:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5710 00000100:00080000:3.0:1787697809.043178:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697809.043182:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697809.043187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697809.043198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134485120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697809.043208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697809.043211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697809.043213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697809.043218:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697809/real 1787697809] req@ffff9d85c0fb1c00 x1874531134484736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697809.043219:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5710 00000100:00080000:3.0:1787697809.043225:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697809.043251:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134484736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697809.043257:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697809.043266:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697809.043267:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697809.043275:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697809.043280:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134484736:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697809.043282:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134485248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697809.043294:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697809.043298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697809.043300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697809.043304:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5710 00000100:00100000:0.0:1787697809.043308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484864:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697809.043308:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697809.043316:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697809.043322:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697809.043334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697809.043337:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134485376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697809.043348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697809/real 1787697809] req@ffff9d85c0fb3800 x1874531134484864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697809.043351:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697809.043359:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697809.043363:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697809.043368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134484864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697809.043369:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5710 00000100:00080000:3.0:1787697809.043375:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697809.043383:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697809.043383:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697809.043391:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787697809.043391:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697809.043398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697809.043407:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134485504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697809.043408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697809.043432:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697809.043445:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697809/real 1787697809] req@ffff9d85c0fb0a80 x1874531134484992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697809.043456:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134484992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697809.043468:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697809.043471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697809.043474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134484992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697809.043479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134485120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697809.043497:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697809.043508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697809/real 1787697809] req@ffff9d85c0fb0e00 x1874531134485120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697809.043519:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134485120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697809.043526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697809.043530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697809.043541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134485120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697809.043550:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134485248:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697809.043566:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697809.043579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697809/real 1787697809] req@ffff9d85c0fb2680 x1874531134485248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697809.043591:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134485248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697809.043598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697809.043602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697809.043648:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134485248:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697809.043669:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134485376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697809.043688:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697809.043699:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697809/real 1787697809] req@ffff9d85c0fb2a00 x1874531134485376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697809.043710:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134485376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697809.043717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697809.043720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697809.043723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134485376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697809.043727:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134485504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697809.043742:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697809.043752:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697809/real 1787697809] req@ffff9d85c0fb3100 x1874531134485504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697809.043761:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134485504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697809.043767:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697809.043770:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697809.043772:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134485504:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697809.619016:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697810.642982:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697811.666959:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697812.690978:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697813.714977:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697814.162930:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697814.162940:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697814.162945:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697814.162961:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5715 00000100:00080000:3.0:1787697814.162972:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697814.162979:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697814.163013:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134485632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697814.163040:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697814.163044:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697814.163046:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697814.163050:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5715 00000100:00080000:3.0:1787697814.163053:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697814.163056:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697814.163063:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697814.163073:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134485760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697814.163084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697814.163088:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134485632:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697814.163089:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697814.163092:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697814.163100:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5715 00000100:00080000:3.0:1787697814.163103:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697814.163107:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697814.163112:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697814.163124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134485888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697814.163134:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697814.163137:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697814.163139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697814.163142:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5715 00000100:00080000:3.0:1787697814.163146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697814.163149:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697814.163168:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697814.163180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697814.163195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134486016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697814.163202:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697814.163205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697814.163206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697814.163209:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5715 00000100:00080000:3.0:1787697814.163212:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697814.163213:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697814/real 1787697814] req@ffff9d85c0fb2d80 x1874531134485632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697814.163217:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697814.163223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697814.163235:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134486144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697814.163242:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134485632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697814.163243:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697814.163249:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697814.163251:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697814.163256:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5715 00000100:00080000:3.0:1787697814.163260:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697814.163263:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697814.163265:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697814.163266:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697814.163273:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697814.163276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134486272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697814.163282:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134485632:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697814.163285:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697814.163290:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697814.163291:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697814.163295:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5715 00000100:00080000:3.0:1787697814.163297:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697814.163299:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697814.163301:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697814.163306:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134486400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697814.163313:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134485760:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697814.163340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697814.163354:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697814/real 1787697814] req@ffff9d85c0fb0380 x1874531134485760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697814.163366:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134485760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697814.163374:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697814.163377:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697814.163381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134485760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697814.163386:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134485888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697814.163405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697814.163415:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697814/real 1787697814] req@ffff9d85c0fb3b80 x1874531134485888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697814.163427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134485888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697814.163440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697814.163443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697814.163446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134485888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697814.163452:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697814.163470:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697814.163480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697814/real 1787697814] req@ffff9d85c0fb3100 x1874531134486016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697814.163490:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134486016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697814.163498:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697814.163501:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697814.163504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697814.163509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134486144:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697814.163525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697814.163549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697814/real 1787697814] req@ffff9d85c0fb2a00 x1874531134486144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697814.163562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134486144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697814.163570:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697814.163580:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697814.163584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134486144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697814.163589:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134486272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697814.163609:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697814.163620:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697814/real 1787697814] req@ffff9d85c0fb2680 x1874531134486272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697814.163633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134486272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697814.163643:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697814.163646:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697814.163649:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134486272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697814.163666:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134486400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697814.163684:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697814.163694:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697814/real 1787697814] req@ffff9d85c0fb0e00 x1874531134486400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697814.163705:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134486400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697814.163712:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697814.163715:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697814.163717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134486400:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697814.739011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697815.762966:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697816.787001:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697817.811008:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697818.834997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697819.282951:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697819.282958:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697819.282961:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697819.282968:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5721 00000100:00080000:2.0:1787697819.282976:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697819.282981:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697819.283009:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134486528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697819.283027:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697819.283029:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697819.283030:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697819.283033:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5721 00000100:00080000:2.0:1787697819.283036:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697819.283039:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697819.283048:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697819.283061:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134486656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697819.283084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134486528:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697819.283085:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697819.283094:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697819.283096:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697819.283101:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5721 00000100:00080000:2.0:1787697819.283105:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697819.283108:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697819.283114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697819.283127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134486784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697819.283136:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697819.283139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697819.283141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697819.283146:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697819.283147:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5721 00000100:00080000:2.0:1787697819.283154:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697819.283158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697819.283163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697819.283176:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134486912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697819.283185:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697819.283189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697819.283192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697819.283193:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697819/real 1787697819] req@ffff9d85babd4000 x1874531134486528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697819.283198:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5721 00000100:00080000:2.0:1787697819.283203:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697819.283207:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697819.283214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697819.283229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134487040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697819.283232:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134486528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697819.283253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697819.283255:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697819.283257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697819.283261:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697819.283262:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697819.283267:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5721 00000100:00100000:0.0:1787697819.283269:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134486528:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697819.283272:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697819.283280:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697819.283285:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697819.283296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134487168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697819.283297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486656:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697819.283313:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697819.283316:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697819.283318:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697819.283322:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5721 00000100:00080000:2.0:1787697819.283326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697819.283329:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697819.283334:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697819.283340:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697819.283347:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697819/real 1787697819] req@ffff9d85babd5c00 x1874531134486656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697819.283354:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134487296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697819.283367:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134486656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.283377:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.283380:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697819.283384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697819.283390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697819.283408:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697819.283418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697819/real 1787697819] req@ffff9d85babd7480 x1874531134486784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.283429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134486784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.283444:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.283447:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697819.283450:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697819.283456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697819.283471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697819.283480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697819/real 1787697819] req@ffff9d85babd5180 x1874531134486912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.283491:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134486912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.283498:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.283508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697819.283512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134486912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697819.283517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487040:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697819.283536:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697819.283547:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697819/real 1787697819] req@ffff9d85babd5f80 x1874531134487040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.283558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134487040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.283563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.283565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697819.283567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487040:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697819.283582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697819.283601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697819.283611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697819/real 1787697819] req@ffff9d85babd4700 x1874531134487168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.283621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134487168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.283625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.283627:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697819.283629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697819.283632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697819.283643:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697819.283655:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697819/real 1787697819] req@ffff9d85babd4a80 x1874531134487296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.283662:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134487296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.283667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.283669:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697819.283670:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487296:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697819.859005:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697820.882978:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697821.907002:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697822.930999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697823.955011:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697824.402944:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697824.402951:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697824.402954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697824.402962:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5726 00000100:00080000:3.0:1787697824.402972:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697824.402998:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697824.403028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134487424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697824.403049:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697824.403051:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697824.403052:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697824.403055:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5726 00000100:00080000:3.0:1787697824.403057:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697824.403060:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697824.403067:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697824.403074:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134487552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697824.403080:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697824.403082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697824.403083:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697824.403085:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5726 00000100:00080000:3.0:1787697824.403087:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697824.403089:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697824.403092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697824.403097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134487680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697824.403103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697824.403105:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697824.403106:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697824.403109:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5726 00000100:00080000:3.0:1787697824.403111:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697824.403113:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697824.403115:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697824.403123:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134487808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697824.403130:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697824.403136:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697824.403137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697824.403138:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134487424:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697824.403144:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5726 00000100:00080000:3.0:1787697824.403153:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697824.403157:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697824.403160:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697824.403167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134487936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697824.403172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697824.403174:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697824.403175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697824.403177:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5726 00000100:00080000:3.0:1787697824.403179:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697824.403181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697824.403185:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697824.403190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134488064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697824.403195:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697824.403197:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697824.403198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697824.403200:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5726 00000100:00080000:3.0:1787697824.403202:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697824.403205:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697824.403208:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697824.403215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134488192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697824.403228:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697824.403275:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697824/real 1787697824] req@ffff9d85c0fb0a80 x1874531134487424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697824.403294:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134487424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697824.403307:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697824.403311:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697824.403316:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134487424:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697824.403324:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134487552:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697824.403342:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697824.403350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697824/real 1787697824] req@ffff9d85c0fb3800 x1874531134487552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697824.403358:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134487552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697824.403363:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697824.403365:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697824.403367:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134487552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697824.403372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134487680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697824.403383:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697824.403392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697824/real 1787697824] req@ffff9d85c0fb1c00 x1874531134487680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697824.403403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134487680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697824.403416:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697824.403419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697824.403422:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134487680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697824.403460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134487808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697824.403480:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697824.403488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697824/real 1787697824] req@ffff9d85c0fb3480 x1874531134487808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697824.403495:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134487808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697824.403501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697824.403503:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697824.403505:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134487808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697824.403509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487936:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697824.403519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697824.403525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697824/real 1787697824] req@ffff9d85c0fb2300 x1874531134487936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697824.403531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134487936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697824.403548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697824.403551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697824.403554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134487936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697824.403561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697824.403580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697824.403588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697824/real 1787697824] req@ffff9d85c0fb1880 x1874531134488064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697824.403595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134488064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697824.403600:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697824.403602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697824.403604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697824.403608:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697824.403619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697824.403624:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697824/real 1787697824] req@ffff9d85c0fb1500 x1874531134488192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697824.403631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134488192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697824.403635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697824.403637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697824.403638:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488192:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697824.979049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697826.003092:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697827.027022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697828.050962:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697829.075060:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697829.523001:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697829.523011:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697829.523015:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697829.523025:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5731 00000100:00080000:1.0:1787697829.523037:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697829.523044:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697829.523083:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134488320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697829.523106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697829.523109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697829.523110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697829.523113:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5731 00000100:00080000:1.0:1787697829.523115:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697829.523118:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697829.523126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697829.523136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134488448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697829.523143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697829.523146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697829.523147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697829.523149:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5731 00000100:00080000:1.0:1787697829.523151:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697829.523154:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697829.523159:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134488320:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787697829.523160:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697829.523170:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134488576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697829.523176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697829.523177:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697829.523178:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697829.523180:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5731 00000100:00080000:1.0:1787697829.523183:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697829.523185:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697829.523187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697829.523194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134488704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697829.523199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697829.523201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697829.523202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697829.523204:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5731 00000100:00080000:1.0:1787697829.523206:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697829.523209:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697829.523212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697829.523218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134488832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697829.523223:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697829.523226:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697829.523227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697829.523230:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5731 00000100:00080000:1.0:1787697829.523232:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697829.523234:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697829.523236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697829.523241:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697829.523252:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134488960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697829.523259:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697829.523261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697829.523262:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697829.523265:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5731 00000100:00080000:1.0:1787697829.523267:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697829.523269:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697829.523272:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697829.523280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134489088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697829.523293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697829/real 1787697829] req@ffff9d85b9d2c380 x1874531134488320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697829.523309:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134488320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697829.523317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697829.523320:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697829.523323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134488320:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697829.523328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134488448:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697829.523344:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697829.523352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697829/real 1787697829] req@ffff9d85b9d2e680 x1874531134488448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697829.523359:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134488448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697829.523364:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697829.523374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697829.523376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134488448:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697829.523397:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134488576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697829.523409:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697829.523416:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697829/real 1787697829] req@ffff9d85b9d2d500 x1874531134488576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697829.523423:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134488576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697829.523432:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697829.523434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697829.523435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134488576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697829.523438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134488704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697829.523447:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697829.523452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697829/real 1787697829] req@ffff9d85b9d2ed80 x1874531134488704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697829.523458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134488704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697829.523463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697829.523465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697829.523466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134488704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697829.523470:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488832:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697829.523479:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697829.523483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697829/real 1787697829] req@ffff9d85b9d2c700 x1874531134488832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697829.523489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134488832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697829.523494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697829.523496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697829.523498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697829.523501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697829.523511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697829.523518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697829/real 1787697829] req@ffff9d85b9d2f800 x1874531134488960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697829.523528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134488960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697829.523535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697829.523538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697829.523541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134488960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697829.523547:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697829.523567:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697829.523577:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697829/real 1787697829] req@ffff9d85b9d2dc00 x1874531134489088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697829.523588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134489088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697829.523596:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697829.523599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697829.523602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489088:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697830.099087:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697831.123027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697832.147018:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697833.170994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697834.194996:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697834.642934:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697834.642944:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697834.642949:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697834.642957:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5736 00000100:00080000:1.0:1787697834.642967:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697834.642974:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697834.643004:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134489216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697834.643025:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697834.643028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697834.643029:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697834.643032:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5736 00000100:00080000:1.0:1787697834.643034:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697834.643037:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697834.643044:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697834.643051:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134489344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697834.643058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697834.643061:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697834.643062:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697834.643066:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5736 00000100:00100000:0.0:1787697834.643067:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134489216:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697834.643067:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697834.643073:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697834.643077:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697834.643084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134489472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697834.643089:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697834.643091:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697834.643093:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697834.643095:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5736 00000100:00080000:1.0:1787697834.643097:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697834.643099:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697834.643102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697834.643108:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134489600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697834.643114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697834.643116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697834.643117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697834.643118:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5736 00000100:00080000:1.0:1787697834.643120:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697834.643124:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697834.643127:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697834.643128:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697834.643137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134489728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697834.643143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697834.643145:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697834.643147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697834.643150:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5736 00000100:00080000:1.0:1787697834.643152:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697834.643156:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697834.643161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697834.643167:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697834/real 1787697834] req@ffff9d85b9d2dc00 x1874531134489216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697834.643173:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134489856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697834.643184:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134489216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697834.643203:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697834.643203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697834.643209:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697834.643210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697834.643213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697834.643216:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134489216:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697834.643220:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5736 00000100:00080000:1.0:1787697834.643224:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697834.643227:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697834.643231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697834.643238:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134489344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697834.643241:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134489984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697834.643269:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697834.643282:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697834/real 1787697834] req@ffff9d85b9d2f800 x1874531134489344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697834.643294:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134489344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697834.643299:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697834.643301:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697834.643303:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134489344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697834.643306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134489472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697834.643319:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697834.643325:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697834/real 1787697834] req@ffff9d85b9d2c700 x1874531134489472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697834.643332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134489472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697834.643348:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697834.643350:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697834.643352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134489472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697834.643355:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134489600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697834.643365:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697834.643370:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697834/real 1787697834] req@ffff9d85b9d2ed80 x1874531134489600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697834.643376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134489600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697834.643381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697834.643382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697834.643384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134489600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697834.643387:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489728:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697834.643395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697834.643400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697834/real 1787697834] req@ffff9d85b9d2d500 x1874531134489728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697834.643406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134489728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697834.643410:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697834.643411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697834.643413:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697834.643416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697834.643424:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697834.643429:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697834/real 1787697834] req@ffff9d85b9d2e680 x1874531134489856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697834.643435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134489856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697834.643439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697834.643440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697834.643442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697834.643453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697834.643461:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697834.643469:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697834/real 1787697834] req@ffff9d85b9d2c380 x1874531134489984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697834.643479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134489984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697834.643486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697834.643489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697834.643492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134489984:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697835.219022:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697836.243013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697837.267037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697838.290988:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697839.315020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697839.762960:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697839.762967:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697839.762970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697839.762977:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5741 00000100:00080000:1.0:1787697839.762986:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697839.762991:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697839.763021:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134490112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697839.763042:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697839.763044:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697839.763046:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697839.763054:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5741 00000100:00080000:1.0:1787697839.763057:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697839.763061:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697839.763070:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697839.763086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134490240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697839.763094:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134490112:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697839.763105:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697839.763108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697839.763109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697839.763113:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5741 00000100:00080000:1.0:1787697839.763116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697839.763120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697839.763125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697839.763135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134490368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697839.763143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697839.763146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697839.763147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697839.763150:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5741 00000100:00080000:1.0:1787697839.763162:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697839.763165:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697839.763169:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697839.763176:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697839.763187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134490496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697839.763194:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697839.763197:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697839.763200:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697839.763204:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5741 00000100:00100000:0.0:1787697839.763207:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697839/real 1787697839] req@ffff9d85b9d2f480 x1874531134490112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697839.763209:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697839.763218:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697839.763223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697839.763226:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134490112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697839.763233:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697839.763237:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697839.763241:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134490112:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697839.763244:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134490624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697839.763253:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697839.763256:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697839.763258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697839.763262:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134490240:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697839.763265:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5741 00000100:00080000:1.0:1787697839.763272:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697839.763276:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697839.763281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697839.763292:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697839.763294:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134490752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697839.763305:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697839.763308:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697839/real 1787697839] req@ffff9d85b9d2fb80 x1874531134490240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697839.763310:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697839.763315:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697839.763322:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5741 00000100:00100000:0.0:1787697839.763327:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134490240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697839.763327:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697839.763334:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697839.763341:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697839.763344:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697839.763350:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787697839.763354:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134490880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697839.763358:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134490240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697839.763370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134490368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697839.763393:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697839.763405:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697839/real 1787697839] req@ffff9d85b9d2e300 x1874531134490368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697839.763416:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134490368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697839.763428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697839.763431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697839.763434:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134490368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697839.763439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134490496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697839.763458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697839.763467:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697839/real 1787697839] req@ffff9d85b9d2ce00 x1874531134490496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697839.763475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134490496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697839.763480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697839.763482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697839.763483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134490496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697839.763487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134490624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697839.763498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697839.763504:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697839/real 1787697839] req@ffff9d85b9d2c000 x1874531134490624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697839.763510:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134490624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697839.763514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697839.763515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697839.763518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134490624:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697839.763530:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134490752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697839.763540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697839.763549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697839/real 1787697839] req@ffff9d85b9d2ca80 x1874531134490752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697839.763560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134490752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697839.763567:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697839.763570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697839.763573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134490752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697839.763578:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134490880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697839.763593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697839.763602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697839/real 1787697839] req@ffff9d85b9d2f480 x1874531134490880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697839.763612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134490880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697839.763617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697839.763618:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697839.763620:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134490880:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697840.338971:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697841.363010:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697842.386951:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697843.411013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697844.435005:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697844.882964:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697844.882973:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697844.882977:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697844.882985:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5746 00000100:00080000:2.0:1787697844.882995:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697844.883013:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697844.883045:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134491008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697844.883066:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697844.883070:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697844.883072:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697844.883076:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5746 00000100:00080000:2.0:1787697844.883079:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697844.883082:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697844.883091:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697844.883102:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134491136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697844.883110:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134491008:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697844.883113:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697844.883118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697844.883119:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697844.883124:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5746 00000100:00080000:2.0:1787697844.883127:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697844.883130:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697844.883136:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697844.883146:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134491264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697844.883154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697844.883158:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697844.883159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697844.883163:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5746 00000100:00080000:2.0:1787697844.883167:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697844.883170:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697844.883175:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697844.883187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134491392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697844.883190:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134491136:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697844.883205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697844.883209:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697844.883211:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697844.883217:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5746 00000100:00100000:0.0:1787697844.883218:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134491264:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787697844.883222:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697844.883229:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697844.883234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697844.883238:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134491392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787697844.883248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134491520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697844.883266:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697844.883270:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697844.883272:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697844.883276:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5746 00000100:00080000:2.0:1787697844.883280:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697844.883285:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134491520:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697844.883286:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697844.883294:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697844.883308:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134491648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697844.883318:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697844.883321:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697844.883323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134491648:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787697844.883324:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697844.883331:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5746 00000100:00080000:2.0:1787697844.883336:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697844.883340:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697844.883345:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697844.883356:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134491776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697844.883386:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134491776:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697845.459005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697846.482991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697847.507008:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697847.955123:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:1.0:1787697847.955162:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697847.955177:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697847.955215:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697847.955224:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697847.955231:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697847.955239:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697847.955247:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787697847.955257:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787697847.955272:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697844/real 1787697847] req@ffff9d85babd5f80 x1874531134491648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.955297:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134491648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.955309:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.955314:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697847.955318:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134491648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697847.955325:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697844/real 1787697847] req@ffff9d85babd4700 x1874531134491520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.955336:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134491520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.955343:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.955346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697847.955349:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134491520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697847.955359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697844/real 1787697847] req@ffff9d85babd4a80 x1874531134491392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.955370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134491392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.955378:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.955381:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697847.955386:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134491392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697847.955392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697844/real 1787697847] req@ffff9d85babd7100 x1874531134491136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.955402:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134491136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.955409:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.955412:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697847.955415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134491136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697847.955421:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697844/real 1787697847] req@ffff9d85babd7b80 x1874531134491264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.955431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134491264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.955438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.955441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697847.955444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134491264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697847.955450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697844/real 1787697847] req@ffff9d85babd4e00 x1874531134491008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.955461:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134491008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.955467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.955470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697847.955473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134491008:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697847.955514:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697844/real 1787697847] req@ffff9d85babd5180 x1874531134491776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.955525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134491776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.955533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.955536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697847.955539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134491776:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697848.530990:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697849.554998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697850.002950:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697850.002958:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697850.002961:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697850.002968:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5751 00000100:00080000:2.0:1787697850.002977:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697850.002983:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697850.003011:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134491904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697850.003028:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697850.003030:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697850.003031:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697850.003034:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5751 00000100:00080000:2.0:1787697850.003036:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697850.003038:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697850.003045:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697850.003055:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134492032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697850.003071:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697850.003073:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134491904:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697850.003075:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697850.003079:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697850.003085:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5751 00000100:00080000:2.0:1787697850.003089:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697850.003092:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697850.003098:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697850.003111:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134492160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697850.003119:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697850.003123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697850.003124:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697850.003128:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5751 00000100:00080000:2.0:1787697850.003131:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697850.003135:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697850.003141:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787697850.003142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697850.003156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134492288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697850.003166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697850.003169:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697850.003170:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697850.003174:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5751 00000100:00080000:2.0:1787697850.003178:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697850.003188:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697850/real 1787697850] req@ffff9d85babd7480 x1874531134491904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697850.003189:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697850.003198:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697850.003211:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134491904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697850.003218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134492416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697850.003224:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697850.003231:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697850.003232:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697850.003237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697850.003242:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134491904:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697850.003242:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697850.003250:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5751 00000100:00080000:2.0:1787697850.003254:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697850.003259:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697850.003264:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697850.003273:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134492544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697850.003286:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492032:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697850.003288:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697850.003295:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697850.003297:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697850.003302:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5751 00000100:00080000:2.0:1787697850.003306:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697850.003310:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697850.003316:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697850.003320:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697850.003332:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134492672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697850.003338:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697850/real 1787697850] req@ffff9d85babd5c00 x1874531134492032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697850.003374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134492032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697850.003385:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697850.003388:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697850.003392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697850.003405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697850.003426:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697850.003438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697850/real 1787697850] req@ffff9d85babd4000 x1874531134492160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697850.003449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134492160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697850.003456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697850.003459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697850.003462:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697850.003468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697850.003491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697850.003502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697850/real 1787697850] req@ffff9d85babd4380 x1874531134492288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697850.003512:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134492288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697850.003518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697850.003521:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697850.003524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697850.003528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134492416:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697850.003573:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697850.003584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697850/real 1787697850] req@ffff9d85babd6300 x1874531134492416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697850.003594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134492416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697850.003606:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697850.003609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697850.003612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134492416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697850.003628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134492544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697850.003647:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697850.003658:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697850/real 1787697850] req@ffff9d85babd6a00 x1874531134492544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697850.003677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134492544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697850.003685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697850.003688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697850.003690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134492544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697850.003695:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134492672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697850.003713:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697850.003730:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697850/real 1787697850] req@ffff9d85babd6680 x1874531134492672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697850.003741:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134492672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697850.003748:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697850.003751:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697850.003754:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134492672:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697850.578971:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697851.603008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697852.626992:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697853.651002:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697854.674986:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697855.122949:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697855.122957:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697855.122959:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697855.122966:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5756 00000100:00080000:3.0:1787697855.122974:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697855.122980:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697855.123006:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134492800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697855.123031:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697855.123035:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697855.123043:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697855.123048:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5756 00000100:00080000:3.0:1787697855.123051:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697855.123056:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697855.123065:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697855.123077:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134492928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697855.123080:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134492800:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697855.123091:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697855.123094:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697855.123095:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697855.123098:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5756 00000100:00080000:3.0:1787697855.123101:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697855.123103:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697855.123107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697855.123119:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134493056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697855.123129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697855.123132:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697855.123134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697855.123137:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697855.123141:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5756 00000100:00080000:3.0:1787697855.123147:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697855.123151:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697855.123156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697855.123167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134493184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697855.123170:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697855/real 1787697855] req@ffff9d85c0fb0700 x1874531134492800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697855.123184:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697855.123187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697855.123188:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697855.123191:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5756 00000100:00080000:3.0:1787697855.123193:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697855.123198:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697855.123199:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134492800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697855.123201:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697855.123212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134493312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697855.123225:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697855.123225:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697855.123231:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697855.123233:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697855.123234:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697855.123241:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5756 00000100:00100000:0.0:1787697855.123244:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134492800:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697855.123245:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697855.123252:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697855.123256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697855.123267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134493440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697855.123269:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492928:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697855.123285:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697855.123289:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697855.123292:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697855.123293:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697855.123299:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5756 00000100:00080000:3.0:1787697855.123304:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697855.123310:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697855.123311:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697855/real 1787697855] req@ffff9d85c0fb0000 x1874531134492928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697855.123316:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697855.123331:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134492928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697855.123333:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134493568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697855.123359:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697855.123363:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697855.123366:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134492928:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697855.123371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697855.123387:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697855.123394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697855/real 1787697855] req@ffff9d85c0fb1f80 x1874531134493056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697855.123401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134493056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697855.123406:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697855.123408:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697855.123410:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697855.123416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697855.123432:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697855.123442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697855/real 1787697855] req@ffff9d85c0fb1500 x1874531134493184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697855.123465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134493184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697855.123474:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697855.123477:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697855.123480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697855.123486:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134493312:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697855.123507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697855.123518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697855/real 1787697855] req@ffff9d85c0fb1880 x1874531134493312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697855.123539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134493312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697855.123547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697855.123550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697855.123553:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134493312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697855.123575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134493440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697855.123590:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697855.123597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697855/real 1787697855] req@ffff9d85c0fb2300 x1874531134493440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697855.123604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134493440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697855.123609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697855.123610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697855.123612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134493440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697855.123615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134493568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697855.123623:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697855.123628:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697855/real 1787697855] req@ffff9d85c0fb3480 x1874531134493568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697855.123634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134493568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697855.123638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697855.123640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697855.123641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134493568:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697855.699028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697856.723004:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697857.747001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697858.771009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697859.794994:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697860.242917:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697860.242926:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697860.242930:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697860.242940:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5762 00000100:00080000:3.0:1787697860.242950:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697860.242956:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697860.242988:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134493696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697860.243012:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697860.243016:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697860.243017:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697860.243023:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5762 00000100:00080000:3.0:1787697860.243026:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697860.243030:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697860.243038:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697860.243049:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134493824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697860.243059:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697860.243061:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134493696:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697860.243063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697860.243067:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697860.243074:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5762 00000100:00080000:3.0:1787697860.243077:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697860.243080:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697860.243085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697860.243096:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134493952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697860.243104:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697860.243107:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697860.243109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697860.243112:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5762 00000100:00080000:3.0:1787697860.243116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697860.243119:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697860.243125:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697860.243127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697860.243139:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134494080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697860.243148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697860.243151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697860.243153:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697860.243157:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5762 00000100:00080000:3.0:1787697860.243161:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697860.243164:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697860/real 1787697860] req@ffff9d85c0fb1c00 x1874531134493696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697860.243176:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697860.243179:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134493696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697860.243183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697860.243196:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697860.243200:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134494208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697860.243201:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697860.243207:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134493696:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697860.243219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697860.243223:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697860.243225:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697860.243229:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5762 00000100:00100000:0.0:1787697860.243232:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493824:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697860.243234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697860.243240:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697860.243251:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697860.243252:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697860.243261:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697860/real 1787697860] req@ffff9d85c0fb3800 x1874531134493824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697860.243266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134494336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697860.243275:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134493824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697860.243283:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697860.243284:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697860.243290:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697860.243290:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697860.243296:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697860.243297:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493824:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697860.243305:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5762 00000100:00100000:0.0:1787697860.243306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493952:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787697860.243311:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697860.243316:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697860.243321:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697860.243323:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697860.243333:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697860/real 1787697860] req@ffff9d85c0fb0a80 x1874531134493952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697860.243338:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134494464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697860.243356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134493952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697860.243379:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697860.243383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697860.243386:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134493952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697860.243393:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697860.243410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697860.243421:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697860/real 1787697860] req@ffff9d85c0fb0e00 x1874531134494080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697860.243432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134494080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697860.243439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697860.243442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697860.243452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697860.243457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134494208:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697860.243474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697860.243484:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697860/real 1787697860] req@ffff9d85c0fb2680 x1874531134494208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697860.243495:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134494208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697860.243502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697860.243505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697860.243508:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134494208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697860.243522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134494336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697860.243536:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697860.243542:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697860/real 1787697860] req@ffff9d85c0fb2a00 x1874531134494336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697860.243549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134494336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697860.243554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697860.243555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697860.243557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134494336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697860.243560:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134494464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697860.243568:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697860.243572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697860/real 1787697860] req@ffff9d85c0fb3100 x1874531134494464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697860.243579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134494464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697860.243583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697860.243584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697860.243586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134494464:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697860.819069:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697861.843005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697862.867049:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697863.891026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697864.915017:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697865.362976:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697865.362991:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697865.362995:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697865.363008:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5767 00000100:00080000:2.0:1787697865.363022:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697865.363032:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697865.363083:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134494592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697865.363114:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697865.363118:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697865.363120:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697865.363133:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5767 00000100:00080000:2.0:1787697865.363138:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697865.363142:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697865.363154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697865.363168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134494720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697865.363181:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697865.363186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697865.363189:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697865.363197:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134494592:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697865.363199:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5767 00000100:00080000:2.0:1787697865.363207:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697865.363212:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697865.363217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697865.363231:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134494848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697865.363240:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697865.363245:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697865.363247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697865.363251:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5767 00000100:00080000:2.0:1787697865.363255:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697865.363268:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697865.363276:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787697865.363288:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697865.363309:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134494976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697865.363318:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697865.363320:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697865.363322:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697865.363327:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5767 00000100:00080000:2.0:1787697865.363332:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697865.363336:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697865.363342:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697865/real 1787697865] req@ffff9d85babd7800 x1874531134494592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697865.363343:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697865.363360:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134495104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697865.363368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134494592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697865.363370:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697865.363380:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697865.363384:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697865.363394:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5767 00000100:00080000:2.0:1787697865.363397:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697865.363403:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697865.363409:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697865.363417:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134495232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697865.363427:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697865.363434:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697865.363438:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697865.363448:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5767 00000100:00080000:2.0:1787697865.363455:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697865.363466:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697865.363470:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697865.363471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697865.363479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787697865.363485:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134495360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697865.363486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134494592:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697865.363518:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494720:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697865.363539:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697865.363548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697865/real 1787697865] req@ffff9d85babd5500 x1874531134494720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697865.363556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134494720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697865.363561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697865.363564:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697865.363565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697865.363569:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697865.363580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697865.363585:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697865/real 1787697865] req@ffff9d85babd6d80 x1874531134494848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697865.363592:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134494848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697865.363596:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697865.363599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697865.363602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697865.363608:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697865.363624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697865.363633:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697865/real 1787697865] req@ffff9d85babd6680 x1874531134494976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697865.363641:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134494976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697865.363646:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697865.363648:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697865.363650:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134494976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697865.363654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134495104:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697865.363666:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697865.363672:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697865/real 1787697865] req@ffff9d85babd6a00 x1874531134495104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697865.363682:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134495104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697865.363690:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697865.363693:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697865.363696:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134495104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697865.363702:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134495232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697865.363720:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697865.363731:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697865/real 1787697865] req@ffff9d85babd6300 x1874531134495232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697865.363742:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134495232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697865.363749:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697865.363752:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697865.363755:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134495232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697865.363760:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134495360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697865.363776:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697865.363783:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697865/real 1787697865] req@ffff9d85babd4380 x1874531134495360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697865.363794:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134495360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697865.363802:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697865.363805:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697865.363808:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134495360:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697865.939029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697866.963027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697867.987068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697869.011064:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697870.035047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697870.482945:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697870.482959:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697870.482964:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697870.482976:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5772 00000100:00080000:2.0:1787697870.482989:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697870.482998:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697870.483044:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134495488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697870.483069:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697870.483072:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697870.483073:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697870.483076:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5772 00000100:00080000:2.0:1787697870.483079:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697870.483082:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697870.483091:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697870.483100:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134495616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697870.483106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697870.483109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697870.483110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697870.483112:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5772 00000100:00080000:2.0:1787697870.483114:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697870.483116:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697870.483119:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697870.483127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134495744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697870.483128:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134495488:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697870.483143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697870.483146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697870.483147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697870.483149:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5772 00000100:00080000:2.0:1787697870.483152:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697870.483154:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697870.483156:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697870.483163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134495872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697870.483172:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697870.483176:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697870.483178:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697870.483181:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5772 00000100:00080000:2.0:1787697870.483185:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697870.483204:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697870.483209:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697870.483220:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697870.483224:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134496000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697870.483232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697870.483235:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697870.483236:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697870.483240:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5772 00000100:00080000:2.0:1787697870.483243:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697870.483247:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697870.483251:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697870.483258:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134496128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697870.483264:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697870.483266:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697870.483267:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697870.483269:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5772 00000100:00080000:2.0:1787697870.483271:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697870.483275:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697870.483278:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697870/real 1787697870] req@ffff9d85babd4000 x1874531134495488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697870.483279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697870.483288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134496256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697870.483309:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134495488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697870.483329:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697870.483334:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697870.483338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134495488:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697870.483345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134495616:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697870.483366:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697870.483375:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697870/real 1787697870] req@ffff9d85babd5c00 x1874531134495616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697870.483382:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134495616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697870.483388:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697870.483390:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697870.483392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134495616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697870.483411:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134495744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697870.483423:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697870.483428:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697870/real 1787697870] req@ffff9d85babd7480 x1874531134495744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697870.483435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134495744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697870.483440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697870.483441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697870.483443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134495744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697870.483447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134495872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697870.483456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697870.483461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697870/real 1787697870] req@ffff9d85babd5180 x1874531134495872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697870.483467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134495872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697870.483472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697870.483474:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697870.483475:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134495872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697870.483478:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496000:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697870.483487:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697870.483491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697870/real 1787697870] req@ffff9d85babd4e00 x1874531134496000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697870.483498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134496000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697870.483502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697870.483504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697870.483505:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697870.483508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697870.483516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697870.483521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697870/real 1787697870] req@ffff9d85babd7b80 x1874531134496128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697870.483528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134496128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697870.483532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697870.483534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697870.483535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697870.483538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697870.483548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697870.483552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697870/real 1787697870] req@ffff9d85babd7100 x1874531134496256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697870.483558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134496256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697870.483562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697870.483563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697870.483565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496256:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697871.059089:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697872.083015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697873.107009:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697874.131058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697875.155040:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697875.602997:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697875.603011:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697875.603016:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697875.603028:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5777 00000100:00080000:2.0:1787697875.603040:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697875.603047:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697875.603098:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134496384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697875.603153:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697875.603156:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697875.603158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697875.603161:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5777 00000100:00080000:2.0:1787697875.603164:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697875.603167:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697875.603176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697875.603188:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134496512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697875.603195:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697875.603198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697875.603199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697875.603207:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5777 00000100:00080000:2.0:1787697875.603211:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697875.603213:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134496384:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697875.603217:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697875.603226:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697875.603249:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134496640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697875.603259:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697875.603261:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697875.603263:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697875.603266:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5777 00000100:00080000:2.0:1787697875.603269:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697875.603272:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697875.603277:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697875.603290:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697875.603297:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134496768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697875.603309:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697875.603312:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697875.603314:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697875.603318:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5777 00000100:00080000:2.0:1787697875.603322:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697875.603327:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697875.603331:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697875.603339:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697875/real 1787697875] req@ffff9d85babd4a80 x1874531134496384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697875.603342:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134496896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697875.603354:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697875.603356:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697875.603357:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697875.603360:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5777 00000100:00080000:2.0:1787697875.603362:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697875.603364:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697875.603366:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697875.603371:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134497024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697875.603377:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697875.603378:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697875.603380:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697875.603382:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5777 00000100:00080000:2.0:1787697875.603384:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697875.603386:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697875.603388:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697875.603404:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134497152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697875.603612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134496384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697875.603629:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697875.603633:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697875.603637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134496384:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697875.603644:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134496512:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697875.603667:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697875.603678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697875/real 1787697875] req@ffff9d85babd4700 x1874531134496512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697875.603690:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134496512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697875.603697:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697875.603700:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697875.603704:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134496512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697875.603729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134496640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697875.603746:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697875.603755:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697875/real 1787697875] req@ffff9d85babd5f80 x1874531134496640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697875.603766:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134496640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697875.603773:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697875.603776:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697875.603780:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134496640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697875.603786:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134496768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697875.603800:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697875.603808:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697875/real 1787697875] req@ffff9d85babd7100 x1874531134496768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697875.603818:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134496768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697875.603826:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697875.603829:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697875.603832:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134496768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697875.603837:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496896:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697875.603878:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697875.603888:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697875/real 1787697875] req@ffff9d85babd7b80 x1874531134496896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697875.603899:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134496896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697875.603907:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697875.603910:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697875.603913:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134496896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697875.603918:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697875.603934:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697875.603942:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697875/real 1787697875] req@ffff9d85babd4e00 x1874531134497024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697875.603953:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134497024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697875.603960:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697875.603963:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697875.603966:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697875.603971:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697875.603985:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697875.603994:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697875/real 1787697875] req@ffff9d85babd5180 x1874531134497152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697875.604003:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134497152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697875.604012:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697875.604014:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697875.604017:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497152:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697876.179100:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697877.203030:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697878.227171:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697879.251002:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697880.275081:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697880.723048:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697880.723062:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697880.723068:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697880.723080:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5782 00000100:00080000:1.0:1787697880.723093:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697880.723101:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697880.723149:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134497280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697880.723176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697880.723180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697880.723185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697880.723190:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5782 00000100:00080000:1.0:1787697880.723193:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697880.723196:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697880.723208:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697880.723220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134497408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697880.723231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697880.723237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697880.723239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697880.723242:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134497280:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697880.723245:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5782 00000100:00080000:1.0:1787697880.723250:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697880.723255:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697880.723261:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697880.723271:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134497536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697880.723280:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697880.723284:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697880.723285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697880.723289:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5782 00000100:00080000:1.0:1787697880.723313:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697880.723317:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697880.723321:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697880.723327:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697880.723334:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134497664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697880.723345:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697880.723348:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697880.723351:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697880.723355:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5782 00000100:00080000:1.0:1787697880.723358:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697880.723363:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697880.723368:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697880.723379:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134497792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697880.723382:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697880/real 1787697880] req@ffff9d85b9d2c380 x1874531134497280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697880.723399:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697880.723403:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697880.723405:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697880.723409:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5782 00000100:00080000:1.0:1787697880.723412:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697880.723416:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134497280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697880.723419:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697880.723425:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697880.723437:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134497920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697880.723439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697880.723447:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697880.723453:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697880.723454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134497280:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697880.723460:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697880.723463:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697880.723467:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5782 00000100:00080000:1.0:1787697880.723470:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697880.723473:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697880.723477:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697880.723483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134497408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697880.723490:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134498048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697880.723506:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697880.723516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697880/real 1787697880] req@ffff9d85b9d2e680 x1874531134497408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697880.723525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134497408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697880.723531:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697880.723533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697880.723535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134497408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697880.723539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134497536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697880.723550:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697880.723558:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697880/real 1787697880] req@ffff9d85b9d2d500 x1874531134497536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697880.723568:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134497536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697880.723575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697880.723578:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697880.723582:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134497536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697880.723588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134497664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697880.723607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697880.723614:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697880/real 1787697880] req@ffff9d85b9d2ed80 x1874531134497664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697880.723622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134497664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697880.723627:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697880.723629:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697880.723630:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134497664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697880.723634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497792:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697880.723645:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697880.723650:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697880/real 1787697880] req@ffff9d85b9d2c700 x1874531134497792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697880.723657:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134497792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697880.723661:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697880.723663:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697880.723664:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697880.723667:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697880.723676:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697880.723681:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697880/real 1787697880] req@ffff9d85b9d2f800 x1874531134497920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697880.723692:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134497920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697880.723700:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697880.723703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697880.723706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134497920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697880.723726:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697880.723742:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697880.723749:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697880/real 1787697880] req@ffff9d85b9d2dc00 x1874531134498048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697880.723756:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134498048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697880.723761:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697880.723763:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697880.723765:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498048:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697881.299061:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697882.323045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697883.347020:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697884.370990:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697885.396161:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697885.842955:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697885.842962:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697885.842965:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697885.842972:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5787 00000100:00080000:3.0:1787697885.842981:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697885.842987:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697885.843020:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134498176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697885.843044:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697885.843047:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697885.843048:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697885.843051:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5787 00000100:00080000:3.0:1787697885.843054:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697885.843056:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697885.843063:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697885.843073:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134498304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697885.843080:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697885.843083:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134498176:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697885.843083:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697885.843087:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697885.843093:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5787 00000100:00080000:3.0:1787697885.843095:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697885.843097:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697885.843100:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697885.843108:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134498432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697885.843114:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697885.843116:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697885.843117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697885.843120:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5787 00000100:00080000:3.0:1787697885.843123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697885.843125:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697885.843127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697885.843134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134498560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697885.843140:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697885.843141:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697885.843146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697885.843148:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697885.843153:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5787 00000100:00080000:3.0:1787697885.843157:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697885.843161:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697885.843165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697885.843176:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134498688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697885.843184:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697885/real 1787697885] req@ffff9d85c0fb3b80 x1874531134498176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697885.843188:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697885.843195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697885.843198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697885.843201:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134498176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697885.843205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5787 00000100:00080000:3.0:1787697885.843212:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697885.843219:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697885.843221:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697885.843224:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697885.843227:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697885.843232:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134498176:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697885.843239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134498816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697885.843246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697885.843248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697885.843250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697885.843251:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134498304:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697885.843255:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5787 00000100:00080000:3.0:1787697885.843259:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697885.843268:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697885.843271:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697885.843274:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697885.843282:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697885/real 1787697885] req@ffff9d85c0fb0380 x1874531134498304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697885.843286:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134498944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697885.843303:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134498304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.843311:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.843315:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697885.843320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134498304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697885.843326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134498432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697885.843346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697885.843356:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697885/real 1787697885] req@ffff9d85c0fb2d80 x1874531134498432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.843366:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134498432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.843370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.843372:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697885.843374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134498432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697885.843378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134498560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697885.843390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697885.843397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697885/real 1787697885] req@ffff9d85c0fb3100 x1874531134498560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.843408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134498560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.843415:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.843418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697885.843421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134498560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697885.843426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498688:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697885.843443:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697885.843450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697885/real 1787697885] req@ffff9d85c0fb2a00 x1874531134498688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.843457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134498688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.843462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.843463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697885.843465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697885.843468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697885.843498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697885.843508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697885/real 1787697885] req@ffff9d85c0fb2680 x1874531134498816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.843520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134498816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.843527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.843530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697885.843533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697885.843550:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697885.843575:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697885.843586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697885/real 1787697885] req@ffff9d85c0fb0e00 x1874531134498944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.843596:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134498944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.843604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.843607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697885.843610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134498944:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697886.419038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697887.443056:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697888.467064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697889.491002:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697890.515053:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697890.962977:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697890.962986:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697890.962990:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697890.963000:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5792 00000100:00080000:3.0:1787697890.963012:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697890.963019:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697890.963059:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134499072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697890.963084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697890.963086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697890.963088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697890.963092:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5792 00000100:00080000:3.0:1787697890.963094:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697890.963096:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697890.963107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697890.963120:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134499200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697890.963130:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697890.963133:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697890.963135:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697890.963139:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5792 00000100:00080000:3.0:1787697890.963143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697890.963147:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697890.963154:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697890.963164:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134499072:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697890.963170:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134499328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697890.963183:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697890.963187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697890.963189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697890.963193:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5792 00000100:00080000:3.0:1787697890.963198:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697890.963202:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697890.963207:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697890.963218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134499456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697890.963225:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697890.963230:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787697890.963233:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697890.963234:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697890.963244:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5792 00000100:00080000:3.0:1787697890.963248:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697890.963251:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697890.963254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697890.963265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134499584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697890.963271:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697890.963275:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697890.963276:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697890.963281:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5792 00000100:00080000:3.0:1787697890.963283:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697890.963287:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697890.963291:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697890.963296:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697890/real 1787697890] req@ffff9d85c0fb0a80 x1874531134499072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697890.963300:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134499712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697890.963360:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697890.963366:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134499072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697890.963377:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697890.963384:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697890.963398:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5792 00000100:00080000:3.0:1787697890.963403:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697890.963405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697890.963422:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697890.963423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787697890.963433:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697890.963440:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134499072:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697890.963448:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134499840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697890.963456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134499200:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697890.963493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697890.963506:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697890/real 1787697890] req@ffff9d85c0fb3800 x1874531134499200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697890.963516:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134499200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697890.963522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697890.963525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697890.963527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134499200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697890.963544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134499328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697890.963558:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697890.963564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697890/real 1787697890] req@ffff9d85c0fb1c00 x1874531134499328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697890.963571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134499328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697890.963575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697890.963577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697890.963579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134499328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697890.963583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134499456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697890.963591:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697890.963596:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697890/real 1787697890] req@ffff9d85c0fb3480 x1874531134499456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697890.963602:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134499456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697890.963607:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697890.963608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697890.963610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134499456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697890.963613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134499584:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697890.963621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697890.963626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697890/real 1787697890] req@ffff9d85c0fb2300 x1874531134499584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697890.963632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134499584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697890.963636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697890.963638:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697890.963641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134499584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697890.963646:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134499712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697890.963663:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697890.963673:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697890/real 1787697890] req@ffff9d85c0fb1880 x1874531134499712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697890.963692:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134499712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697890.963698:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697890.963701:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697890.963704:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134499712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697890.963710:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134499840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697890.963729:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697890.963740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697890/real 1787697890] req@ffff9d85c0fb1500 x1874531134499840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697890.963750:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134499840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697890.963755:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697890.963757:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697890.963759:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134499840:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697891.539041:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697892.563076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697893.587042:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697894.611053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697895.635040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697896.082977:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697896.082986:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697896.082990:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697896.083002:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5797 00000100:00080000:3.0:1787697896.083016:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697896.083026:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697896.083068:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134499968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697896.083097:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697896.083100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697896.083101:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697896.083107:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5797 00000100:00080000:3.0:1787697896.083110:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697896.083113:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697896.083124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697896.083133:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134500096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697896.083138:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134499968:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697896.083146:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697896.083149:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697896.083149:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697896.083152:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5797 00000100:00080000:3.0:1787697896.083155:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697896.083157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697896.083160:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697896.083167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134500224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697896.083173:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697896.083177:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697896.083179:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697896.083183:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5797 00000100:00080000:3.0:1787697896.083187:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697896.083191:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697896.083195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697896.083207:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697896.083211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134500352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697896.083222:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697896.083225:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697896.083227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697896.083230:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5797 00000100:00080000:3.0:1787697896.083233:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697896.083237:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697896.083240:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697896.083250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134500480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697896.083253:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697896/real 1787697896] req@ffff9d85c0fb1f80 x1874531134499968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697896.083269:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697896.083274:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697896.083277:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134499968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697896.083278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697896.083288:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5797 00000100:00080000:3.0:1787697896.083294:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697896.083300:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697896.083307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697896.083308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697896.083314:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697896.083318:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134500608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697896.083323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134499968:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697896.083328:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697896.083332:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697896.083333:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697896.083336:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5797 00000100:00080000:3.0:1787697896.083339:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697896.083341:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697896.083343:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697896.083349:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134500736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697896.083356:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500096:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697896.083396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697896.083410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697896/real 1787697896] req@ffff9d85c0fb0000 x1874531134500096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697896.083423:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134500096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697896.083431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697896.083434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697896.083437:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697896.083443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697896.083464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697896.083475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697896/real 1787697896] req@ffff9d85c0fb0700 x1874531134500224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697896.083487:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134500224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697896.083494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697896.083498:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697896.083501:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697896.083507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697896.083525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697896.083533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697896/real 1787697896] req@ffff9d85c0fb1500 x1874531134500352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697896.083540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134500352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697896.083545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697896.083547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697896.083549:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697896.083552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134500480:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697896.083562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697896.083568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697896/real 1787697896] req@ffff9d85c0fb1880 x1874531134500480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697896.083574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134500480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697896.083578:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697896.083580:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697896.083581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134500480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697896.083584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134500608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697896.083593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697896.083597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697896/real 1787697896] req@ffff9d85c0fb2300 x1874531134500608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697896.083604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134500608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697896.083609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697896.083610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697896.083612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134500608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697896.083622:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134500736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697896.083631:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697896.083636:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697896/real 1787697896] req@ffff9d85c0fb3480 x1874531134500736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697896.083642:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134500736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697896.083647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697896.083648:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697896.083650:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134500736:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697896.659038:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697897.683038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697898.706960:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697899.730995:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697900.755009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697901.202953:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697901.202963:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697901.202966:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697901.202974:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5802 00000100:00080000:3.0:1787697901.202982:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697901.202987:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697901.203016:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134500864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697901.203037:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697901.203040:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697901.203041:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697901.203044:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5802 00000100:00080000:3.0:1787697901.203047:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697901.203051:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697901.203059:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697901.203072:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134500992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697901.203081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697901.203085:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697901.203086:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697901.203090:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5802 00000100:00080000:3.0:1787697901.203093:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697901.203095:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697901.203099:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697901.203101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134500864:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697901.203110:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134501120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697901.203120:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697901.203124:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697901.203125:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697901.203129:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5802 00000100:00080000:3.0:1787697901.203132:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697901.203136:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697901.203141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697901.203164:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134501248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697901.203167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697901.203175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697901.203178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697901.203179:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697901.203183:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5802 00000100:00080000:3.0:1787697901.203186:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697901.203189:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697901.203192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697901.203200:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134501376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697901.203206:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697901.203210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697901.203211:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697901/real 1787697901] req@ffff9d85c0fb1c00 x1874531134500864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697901.203213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697901.203222:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5802 00000100:00080000:3.0:1787697901.203228:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697901.203232:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697901.203235:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134500864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697901.203239:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697901.203253:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134501504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697901.203254:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697901.203262:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697901.203269:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134500864:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697901.203272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697901.203277:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697901.203278:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500992:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697901.203279:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697901.203287:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5802 00000100:00080000:3.0:1787697901.203291:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697901.203295:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697901.203300:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697901.203303:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697901.203312:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134501632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697901.203318:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697901/real 1787697901] req@ffff9d85c0fb3800 x1874531134500992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697901.203332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134500992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697901.203338:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697901.203342:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697901.203345:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134500992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697901.203373:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134501120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697901.203390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697901.203399:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697901/real 1787697901] req@ffff9d85c0fb0a80 x1874531134501120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697901.203408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134501120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697901.203415:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697901.203418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697901.203421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134501120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697901.203427:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134501248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697901.203441:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697901.203449:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697901/real 1787697901] req@ffff9d85c0fb0e00 x1874531134501248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697901.203458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134501248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697901.203465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697901.203467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697901.203470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134501248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697901.203474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134501376:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697901.203489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697901.203497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697901/real 1787697901] req@ffff9d85c0fb2680 x1874531134501376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697901.203506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134501376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697901.203512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697901.203514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697901.203517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134501376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697901.203521:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134501504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697901.203535:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697901.203542:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697901/real 1787697901] req@ffff9d85c0fb2a00 x1874531134501504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697901.203552:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134501504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697901.203558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697901.203560:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697901.203562:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134501504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697901.203567:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134501632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697901.203579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697901.203586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697901/real 1787697901] req@ffff9d85c0fb3100 x1874531134501632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697901.203594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134501632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697901.203600:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697901.203602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697901.203605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134501632:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697901.779025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697902.802983:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697903.827013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697904.851023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697905.875100:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697906.322982:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697906.322997:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697906.323004:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697906.323018:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5808 00000100:00080000:3.0:1787697906.323032:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697906.323042:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697906.323094:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134501760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697906.323132:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697906.323135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697906.323137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697906.323141:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5808 00000100:00080000:3.0:1787697906.323144:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697906.323147:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697906.323157:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697906.323168:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134501888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697906.323176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697906.323178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697906.323180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697906.323183:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5808 00000100:00080000:3.0:1787697906.323184:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697906.323187:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697906.323190:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697906.323198:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134501760:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697906.323199:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134502016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697906.323214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697906.323217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697906.323218:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697906.323220:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5808 00000100:00080000:3.0:1787697906.323222:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697906.323224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697906.323228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697906.323238:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134502144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697906.323260:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697906.323264:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697906.323266:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697906.323270:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5808 00000100:00080000:3.0:1787697906.323276:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697906.323281:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697906.323286:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697906.323292:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697906.323304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134502272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697906.323316:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697906.323320:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697906.323322:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697906.323327:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5808 00000100:00080000:3.0:1787697906.323332:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697906.323336:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697906.323341:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697906.323350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697906/real 1787697906] req@ffff9d85c0fb2d80 x1874531134501760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697906.323352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134502400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697906.323385:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697906.323390:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697906.323393:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697906.323396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134501760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697906.323401:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5808 00000100:00080000:3.0:1787697906.323410:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697906.323419:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697906.323424:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697906.323426:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697906.323434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697906.323440:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134501760:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697906.323445:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134502528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697906.323454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134501888:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697906.323488:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697906.323504:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697906/real 1787697906] req@ffff9d85c0fb0380 x1874531134501888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697906.323517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134501888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697906.323527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697906.323530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697906.323534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134501888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697906.323563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697906.323617:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697906.323631:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697906/real 1787697906] req@ffff9d85c0fb3b80 x1874531134502016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697906.323645:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134502016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697906.323654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697906.323657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697906.323662:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697906.323669:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697906.323690:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697906.323702:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697906/real 1787697906] req@ffff9d85c0fb3100 x1874531134502144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697906.323715:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134502144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697906.323724:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697906.323728:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697906.323731:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697906.323738:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134502272:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697906.323759:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697906.323772:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697906/real 1787697906] req@ffff9d85c0fb2a00 x1874531134502272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697906.323785:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134502272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697906.323793:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697906.323797:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697906.323800:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134502272:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697906.323807:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134502400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697906.323827:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697906.323838:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697906/real 1787697906] req@ffff9d85c0fb2680 x1874531134502400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697906.323882:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134502400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697906.323890:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697906.323894:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697906.323897:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134502400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697906.323903:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134502528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697906.323925:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697906.323936:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697906/real 1787697906] req@ffff9d85c0fb0e00 x1874531134502528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697906.323949:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134502528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697906.323958:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697906.323962:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697906.323965:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134502528:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697906.899090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697907.923042:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697908.947011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697909.971069:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697910.994975:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697911.443070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697911.443084:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697911.443089:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697911.443103:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5813 00000100:00080000:1.0:1787697911.443118:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697911.443127:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697911.443179:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134502656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697911.443217:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697911.443221:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697911.443223:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697911.443228:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5813 00000100:00080000:1.0:1787697911.443233:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697911.443237:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697911.443248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697911.443262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134502784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697911.443270:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697911.443272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697911.443274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697911.443277:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5813 00000100:00080000:1.0:1787697911.443279:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697911.443287:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134502656:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697911.443292:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697911.443300:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697911.443311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134502912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697911.443321:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697911.443325:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697911.443328:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697911.443331:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5813 00000100:00080000:1.0:1787697911.443335:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697911.443340:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697911.443344:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697911.443352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134503040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697911.443366:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697911.443369:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697911.443370:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697911.443373:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5813 00000100:00080000:1.0:1787697911.443375:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697911.443378:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697911.443381:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697911.443387:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134503168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697911.443393:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697911.443396:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697911.443397:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697911.443399:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5813 00000100:00080000:1.0:1787697911.443401:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697911.443404:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697911.443406:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697911.443413:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134503296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697911.443421:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697911.443424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502784:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697911.443425:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697911.443428:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697911.443435:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5813 00000100:00080000:1.0:1787697911.443439:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697911.443443:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697911.443448:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697911.443462:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134503424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697911.443473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697911.443522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697911.443548:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134503168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697911.443574:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134503296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697911.443602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134503424:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697912.019043:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697913.043000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697914.067008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697914.515085:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697914.515128:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697914.515144:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697914.515187:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697914.515197:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697914.515204:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697914.515211:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697914.515218:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697914.515226:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787697914.515316:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697911/real 1787697914] req@ffff9d85b9d2c380 x1874531134503424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697914.515339:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134503424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697914.515350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697914.515355:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697914.515359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134503424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697914.515365:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697911/real 1787697914] req@ffff9d85b9d2c700 x1874531134502912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697914.515374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134502912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697914.515380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697914.515383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697914.515385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697914.515391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697911/real 1787697914] req@ffff9d85b9d2ed80 x1874531134503040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697914.515401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134503040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697914.515421:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697914.515424:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697914.515427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697914.515433:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697911/real 1787697914] req@ffff9d85b9d2d500 x1874531134503168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697914.515477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134503168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697914.515485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697914.515488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697914.515491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134503168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697914.515497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697911/real 1787697914] req@ffff9d85b9d2e680 x1874531134503296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697914.515507:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134503296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697914.515514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697914.515517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697914.515521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134503296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697914.515526:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697911/real 1787697914] req@ffff9d85b9d2dc00 x1874531134502656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697914.515536:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134502656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697914.515544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697914.515547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697914.515550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134502656:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697914.515556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697911/real 1787697914] req@ffff9d85b9d2f800 x1874531134502784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697914.515567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134502784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697914.515575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697914.515578:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697914.515581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134502784:192.168.201.137@tcp:38:kworker.0 00010000:00010000:3.0:1787697915.091023:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697916.114964:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697916.562930:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697916.562938:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697916.562940:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697916.562947:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5818 00000100:00080000:1.0:1787697916.562956:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697916.562961:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697916.562993:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134503552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697916.563016:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697916.563019:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697916.563020:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697916.563023:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5818 00000100:00080000:1.0:1787697916.563025:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697916.563026:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697916.563033:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697916.563040:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134503680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697916.563045:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697916.563047:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697916.563048:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697916.563050:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5818 00000100:00080000:1.0:1787697916.563052:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697916.563054:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697916.563056:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697916.563062:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134503808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697916.563074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697916.563077:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697916.563079:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697916.563082:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5818 00000100:00080000:1.0:1787697916.563086:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697916.563089:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697916.563094:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697916.563107:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134503936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697916.563117:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134503552:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697916.563121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697916.563132:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697916.563134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697916.563138:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5818 00000100:00080000:1.0:1787697916.563142:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697916.563147:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697916.563151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697916.563165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134504064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697916.563175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697916.563178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697916.563180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697916.563184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5818 00000100:00080000:1.0:1787697916.563219:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697916.563223:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697916.563227:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697916.563235:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697916.563243:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134504192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697916.563266:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697916.563269:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697916.563270:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697916.563274:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5818 00000100:00080000:1.0:1787697916.563278:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697916.563281:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697916.563286:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697916.563302:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134504320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697916.563310:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697916/real 1787697916] req@ffff9d85b9d2f480 x1874531134503552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697916.563339:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134503552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697916.563352:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697916.563356:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697916.563361:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134503552:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697916.563370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503680:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697916.563399:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697916.563427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697916/real 1787697916] req@ffff9d85b9d2ca80 x1874531134503680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697916.563441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134503680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697916.563450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697916.563453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697916.563457:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697916.563463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697916.563485:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697916.563497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697916/real 1787697916] req@ffff9d85b9d2c000 x1874531134503808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697916.563509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134503808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697916.563517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697916.563520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697916.563524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697916.563556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697916.563577:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697916.563591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697916/real 1787697916] req@ffff9d85b9d2ce00 x1874531134503936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697916.563608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134503936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697916.563617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697916.563620:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697916.563624:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134503936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697916.563630:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504064:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697916.563651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697916.563663:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697916/real 1787697916] req@ffff9d85b9d2e300 x1874531134504064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697916.563675:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134504064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697916.563683:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697916.563687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697916.563690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697916.563698:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697916.563719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697916.563729:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697916/real 1787697916] req@ffff9d85b9d2fb80 x1874531134504192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697916.563741:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134504192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697916.563755:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697916.563759:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697916.563762:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697916.563768:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697916.563785:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697916.563794:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697916/real 1787697916] req@ffff9d85b9d2f800 x1874531134504320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697916.563804:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134504320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697916.563813:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697916.563817:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697916.563820:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504320:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697917.139136:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697918.162972:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697919.187021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697920.212488:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697921.235015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697921.682927:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697921.682933:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697921.682936:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697921.682943:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5823 00000100:00080000:3.0:1787697921.682952:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697921.682958:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697921.682985:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134504448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697921.683005:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697921.683008:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697921.683009:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697921.683013:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5823 00000100:00080000:3.0:1787697921.683017:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697921.683020:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697921.683028:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697921.683039:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134504576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697921.683047:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697921.683049:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134504448:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697921.683050:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697921.683052:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697921.683055:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5823 00000100:00080000:3.0:1787697921.683058:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697921.683061:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697921.683065:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697921.683072:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134504704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697921.683079:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697921.683082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697921.683083:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697921.683086:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5823 00000100:00080000:3.0:1787697921.683088:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697921.683091:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697921.683094:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697921.683101:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697921.683102:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134504832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697921.683109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697921.683112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697921.683113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697921.683116:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5823 00000100:00080000:3.0:1787697921.683119:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697921.683122:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697921.683125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697921.683133:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134504960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697921.683134:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697921/real 1787697921] req@ffff9d85c0fb0a80 x1874531134504448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697921.683141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697921.683144:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697921.683146:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697921.683149:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134504448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697921.683149:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5823 00000100:00080000:3.0:1787697921.683151:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697921.683155:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697921.683158:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697921.683159:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697921.683161:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697921.683165:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134504448:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697921.683166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134505088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697921.683173:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697921.683175:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697921.683177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697921.683179:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5823 00000100:00080000:3.0:1787697921.683181:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697921.683184:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134504576:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697921.683184:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697921.683188:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697921.683195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134505216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697921.683202:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697921.683213:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697921/real 1787697921] req@ffff9d85c0fb3800 x1874531134504576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697921.683224:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134504576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697921.683232:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697921.683235:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697921.683238:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134504576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697921.683243:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134504704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697921.683257:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697921.683264:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697921/real 1787697921] req@ffff9d85c0fb1c00 x1874531134504704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697921.683271:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134504704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697921.683276:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697921.683277:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697921.683279:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134504704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697921.683283:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134504832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697921.683297:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697921.683306:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697921/real 1787697921] req@ffff9d85c0fb3480 x1874531134504832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697921.683328:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134504832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697921.683336:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697921.683339:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697921.683342:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134504832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697921.683347:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504960:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697921.683366:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697921.683377:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697921/real 1787697921] req@ffff9d85c0fb2300 x1874531134504960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697921.683383:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134504960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697921.683388:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697921.683390:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697921.683392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134504960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697921.683395:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697921.683406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697921.683411:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697921/real 1787697921] req@ffff9d85c0fb1880 x1874531134505088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697921.683417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134505088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697921.683425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697921.683427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697921.683428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697921.683439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697921.683448:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697921.683453:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697921/real 1787697921] req@ffff9d85c0fb1500 x1874531134505216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697921.683459:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134505216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697921.683463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697921.683465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697921.683466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505216:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697922.259056:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697923.283027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697924.307023:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697925.331011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697926.355022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697926.802938:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697926.802946:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697926.802949:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697926.802957:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5828 00000100:00080000:3.0:1787697926.802965:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697926.802970:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697926.802999:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134505344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697926.803021:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697926.803024:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697926.803025:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697926.803028:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5828 00000100:00080000:3.0:1787697926.803030:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697926.803032:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697926.803039:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697926.803047:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134505472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697926.803054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697926.803056:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697926.803057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697926.803061:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5828 00000100:00080000:3.0:1787697926.803063:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697926.803064:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134505344:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697926.803066:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697926.803073:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697926.803079:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134505600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697926.803084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697926.803086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697926.803088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697926.803090:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5828 00000100:00080000:3.0:1787697926.803091:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697926.803094:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697926.803096:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697926.803105:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134505728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697926.803110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697926.803112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697926.803113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697926.803115:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5828 00000100:00080000:3.0:1787697926.803116:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697926.803121:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697926.803123:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697926.803124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697926.803132:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134505856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697926.803139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697926.803141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697926.803142:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697926.803144:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5828 00000100:00080000:3.0:1787697926.803146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697926.803148:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697926.803151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697926.803159:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697926/real 1787697926] req@ffff9d85c0fb0700 x1874531134505344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697926.803160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134505984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697926.803190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697926.803192:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697926.803194:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697926.803197:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134505344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697926.803200:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5828 00000100:00080000:3.0:1787697926.803204:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697926.803210:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697926.803215:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697926.803216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697926.803223:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697926.803226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134506112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697926.803238:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134505344:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697926.803260:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134505472:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697926.803279:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697926.803286:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697926/real 1787697926] req@ffff9d85c0fb0000 x1874531134505472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697926.803293:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134505472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697926.803299:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697926.803301:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697926.803303:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134505472:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697926.803306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134505600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697926.803315:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697926.803320:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697926/real 1787697926] req@ffff9d85c0fb1f80 x1874531134505600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697926.803327:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134505600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697926.803335:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697926.803338:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697926.803340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134505600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697926.803346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134505728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697926.803363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697926.803373:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697926/real 1787697926] req@ffff9d85c0fb1500 x1874531134505728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697926.803384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134505728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697926.803392:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697926.803395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697926.803398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134505728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697926.803403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505856:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697926.803420:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697926.803427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697926/real 1787697926] req@ffff9d85c0fb1880 x1874531134505856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697926.803434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134505856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697926.803439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697926.803440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697926.803442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697926.803445:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697926.803454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697926.803459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697926/real 1787697926] req@ffff9d85c0fb2300 x1874531134505984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697926.803465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134505984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697926.803473:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697926.803475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697926.803477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134505984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697926.803479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134506112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697926.803487:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697926.803491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697926/real 1787697926] req@ffff9d85c0fb3480 x1874531134506112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697926.803497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134506112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697926.803501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697926.803503:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697926.803504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134506112:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697927.379020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697928.402962:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697929.427025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697930.451038:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697931.475053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697931.922921:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697931.922928:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697931.922931:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697931.922938:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5833 00000100:00080000:1.0:1787697931.922946:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697931.922952:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697931.922978:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134506240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697931.923002:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697931.923005:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697931.923006:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697931.923009:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5833 00000100:00080000:1.0:1787697931.923011:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697931.923013:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697931.923020:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697931.923029:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134506368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697931.923036:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697931.923039:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134506240:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697931.923040:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697931.923042:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697931.923049:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5833 00000100:00080000:1.0:1787697931.923051:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697931.923053:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697931.923056:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697931.923069:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134506496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697931.923076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697931.923078:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697931.923080:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697931.923082:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5833 00000100:00080000:1.0:1787697931.923087:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697931.923089:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697931.923093:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697931.923097:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697931.923104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134506624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697931.923111:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697931.923113:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697931.923115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697931.923118:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5833 00000100:00080000:1.0:1787697931.923122:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697931.923127:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697931.923131:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697931.923134:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697931/real 1787697931] req@ffff9d85b9d2dc00 x1874531134506240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697931.923166:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134506752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697931.923168:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134506240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697931.923191:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697931.923193:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697931.923199:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697931.923202:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697931.923203:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134506240:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697931.923206:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697931.923215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5833 00000100:00080000:1.0:1787697931.923219:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697931.923223:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697931.923227:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697931.923235:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134506368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697931.923238:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134506880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697931.923248:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697931.923250:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697931.923251:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697931.923254:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5833 00000100:00080000:1.0:1787697931.923263:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697931.923267:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697931.923268:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697931.923270:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697931.923287:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697931/real 1787697931] req@ffff9d85b9d2e680 x1874531134506368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697931.923289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134507008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697931.923316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134506368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697931.923326:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697931.923331:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697931.923335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134506368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697931.923341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134506496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697931.923361:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697931.923372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697931/real 1787697931] req@ffff9d85b9d2d500 x1874531134506496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697931.923388:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134506496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697931.923397:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697931.923400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697931.923403:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134506496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697931.923408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134506624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697931.923427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697931.923438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697931/real 1787697931] req@ffff9d85b9d2ed80 x1874531134506624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697931.923449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134506624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697931.923457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697931.923460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697931.923463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134506624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697931.923468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134506752:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697931.923487:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697931.923497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697931/real 1787697931] req@ffff9d85b9d2c700 x1874531134506752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697931.923507:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134506752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697931.923514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697931.923516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697931.923519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134506752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697931.923536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134506880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697931.923553:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697931.923560:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697931/real 1787697931] req@ffff9d85b9d2c380 x1874531134506880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697931.923567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134506880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697931.923576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697931.923577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697931.923580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134506880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697931.923586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697931.923601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697931.923611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697931/real 1787697931] req@ffff9d85b9d2dc00 x1874531134507008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697931.923622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134507008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697931.923630:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697931.923633:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697931.923635:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507008:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697932.499050:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697933.523028:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697934.547027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697935.571006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697936.595074:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697937.043014:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697937.043027:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697937.043033:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697937.043046:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5838 00000100:00080000:2.0:1787697937.043087:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697937.043097:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697937.043145:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134507136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697937.043179:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697937.043183:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697937.043185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697937.043190:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5838 00000100:00080000:2.0:1787697937.043193:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697937.043198:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697937.043209:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697937.043226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134507264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697937.043239:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697937.043245:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697937.043247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134507136:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697937.043248:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697937.043257:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5838 00000100:00080000:2.0:1787697937.043263:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697937.043267:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697937.043273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697937.043288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134507392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697937.043298:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697937.043302:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697937.043305:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697937.043309:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5838 00000100:00080000:2.0:1787697937.043315:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697937.043320:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697937.043325:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697937.043327:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787697937.043344:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134507520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697937.043354:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697937.043358:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697937.043361:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697937.043366:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5838 00000100:00080000:2.0:1787697937.043371:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697937.043376:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697937.043382:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697937/real 1787697937] req@ffff9d85babd7480 x1874531134507136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697937.043383:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697937.043402:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134507648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697937.043411:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134507136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697937.043416:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697937.043427:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697937.043434:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697937.043444:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5838 00000100:00080000:0.0:1787697937.043447:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697937.043449:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697937.043455:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787697937.043459:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697937.043467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134507136:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787697937.043470:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697937.043493:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134507776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697937.043505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134507264:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787697937.043507:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697937.043518:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697937.043521:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697937.043527:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5838 00000100:00080000:2.0:1787697937.043531:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697937.043536:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697937.043543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787697937.043544:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697937.043560:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697937/real 1787697937] req@ffff9d85babd5c00 x1874531134507264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697937.043566:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134507904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697937.043614:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134507264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697937.043624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697937.043628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697937.043632:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134507264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697937.043641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134507392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697937.043662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697937.043673:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697937/real 1787697937] req@ffff9d85babd4000 x1874531134507392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697937.043684:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134507392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697937.043691:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697937.043694:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697937.043697:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134507392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697937.043703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134507520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697937.043723:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697937.043734:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697937/real 1787697937] req@ffff9d85babd4380 x1874531134507520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697937.043745:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134507520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697937.043752:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697937.043755:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697937.043758:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134507520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697937.043764:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507648:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697937.043782:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697937.043793:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697937/real 1787697937] req@ffff9d85babd6300 x1874531134507648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697937.043805:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134507648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697937.043812:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697937.043815:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697937.043818:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697937.043835:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697937.043888:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697937.043900:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697937/real 1787697937] req@ffff9d85babd6a00 x1874531134507776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697937.043911:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134507776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697937.043925:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697937.043928:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697937.043932:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697937.043938:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697937.043958:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697937.043969:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697937/real 1787697937] req@ffff9d85babd6680 x1874531134507904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697937.043981:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134507904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697937.043989:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697937.043992:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697937.043995:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134507904:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697937.619009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697938.643011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697939.666962:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697940.690958:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697941.715086:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697942.163001:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697942.163013:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697942.163019:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697942.163032:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5843 00000100:00080000:1.0:1787697942.163044:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697942.163052:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697942.163099:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134508032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697942.163132:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697942.163136:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697942.163138:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697942.163142:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5843 00000100:00080000:1.0:1787697942.163146:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697942.163150:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697942.163161:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697942.163175:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134508032:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697942.163177:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134508160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697942.163190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697942.163194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697942.163196:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697942.163200:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5843 00000100:00080000:1.0:1787697942.163203:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697942.163208:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697942.163214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697942.163226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134508288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697942.163234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697942.163238:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697942.163240:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697942.163243:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5843 00000100:00080000:1.0:1787697942.163246:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697942.163249:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697942.163254:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787697942.163255:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697942.163267:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134508416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697942.163277:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697942.163281:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697942.163282:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697942.163285:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5843 00000100:00080000:1.0:1787697942.163288:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697942.163292:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697942.163295:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697942.163304:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134508544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697942.163309:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697942/real 1787697942] req@ffff9d85b9d2f800 x1874531134508032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697942.163312:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697942.163319:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697942.163321:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697942.163329:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5843 00000100:00080000:1.0:1787697942.163333:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697942.163336:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697942.163339:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697942.163345:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134508032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697942.163347:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134508672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697942.163374:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697942.163377:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697942.163378:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697942.163379:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697942.163386:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697942.163387:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5843 00000100:00080000:1.0:1787697942.163392:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697942.163394:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134508032:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697942.163397:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697942.163403:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697942.163410:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134508800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697942.163425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134508160:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697942.163447:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697942.163460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697942/real 1787697942] req@ffff9d85b9d2fb80 x1874531134508160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697942.163474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134508160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697942.163483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697942.163486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697942.163489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134508160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697942.163495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134508288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697942.163516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697942.163527:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697942/real 1787697942] req@ffff9d85b9d2e300 x1874531134508288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697942.163538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134508288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697942.163543:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697942.163545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697942.163547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134508288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697942.163551:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134508416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697942.163564:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697942.163571:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697942/real 1787697942] req@ffff9d85b9d2ce00 x1874531134508416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697942.163578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134508416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697942.163583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697942.163584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697942.163586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134508416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697942.163589:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134508544:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697942.163606:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697942.163612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697942/real 1787697942] req@ffff9d85b9d2c000 x1874531134508544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697942.163619:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134508544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697942.163628:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697942.163630:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697942.163633:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134508544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697942.163638:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134508672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697942.163655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697942.163665:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697942/real 1787697942] req@ffff9d85b9d2ca80 x1874531134508672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697942.163677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134508672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697942.163690:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697942.163694:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697942.163698:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134508672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697942.163703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134508800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697942.163721:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697942.163731:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697942/real 1787697942] req@ffff9d85b9d2f480 x1874531134508800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697942.163742:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134508800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697942.163750:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697942.163753:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697942.163756:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134508800:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697942.739076:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697943.763038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697944.787067:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697945.811047:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697946.835097:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697947.282970:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697947.282978:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697947.282981:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697947.282991:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5849 00000100:00080000:1.0:1787697947.283000:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697947.283008:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697947.283044:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134508928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697947.283070:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697947.283074:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697947.283076:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697947.283080:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5849 00000100:00080000:1.0:1787697947.283084:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697947.283087:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697947.283096:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697947.283106:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134509056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697947.283114:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697947.283116:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697947.283117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697947.283120:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5849 00000100:00080000:1.0:1787697947.283122:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697947.283125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697947.283128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697947.283135:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134509184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697947.283140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697947.283143:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697947.283143:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697947.283146:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5849 00000100:00080000:1.0:1787697947.283149:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697947.283151:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697947.283154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697947.283164:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134509312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697947.283168:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134508928:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697947.283171:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697947.283180:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697947.283187:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697947.283191:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5849 00000100:00080000:1.0:1787697947.283192:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697947.283194:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697947.283196:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697947.283205:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134509440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697947.283211:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697947.283213:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697947.283215:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697947.283217:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5849 00000100:00080000:1.0:1787697947.283219:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697947.283221:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697947.283223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697947.283232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134509568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697947.283244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697947.283245:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697947.283247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697947.283249:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5849 00000100:00080000:1.0:1787697947.283260:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697947.283262:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697947.283265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697947.283277:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134509696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697947.283280:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697947.283349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697947/real 1787697947] req@ffff9d85b9d2f480 x1874531134508928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697947.283372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134508928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697947.283384:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697947.283389:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697947.283393:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134508928:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697947.283403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509056:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697947.283428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697947.283441:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697947/real 1787697947] req@ffff9d85b9d2ca80 x1874531134509056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697947.283452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134509056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697947.283460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697947.283463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697947.283467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697947.283473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697947.283493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697947.283504:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697947/real 1787697947] req@ffff9d85b9d2c000 x1874531134509184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697947.283515:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134509184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697947.283522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697947.283525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697947.283528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697947.283557:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697947.283577:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697947.283589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697947/real 1787697947] req@ffff9d85b9d2ce00 x1874531134509312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697947.283600:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134509312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697947.283608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697947.283610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697947.283614:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697947.283620:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134509440:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697947.283639:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697947.283649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697947/real 1787697947] req@ffff9d85b9d2e300 x1874531134509440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697947.283661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134509440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697947.283668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697947.283671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697947.283674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134509440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697947.283680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134509568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697947.283698:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697947.283709:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697947/real 1787697947] req@ffff9d85b9d2fb80 x1874531134509568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697947.283719:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134509568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697947.283732:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697947.283735:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697947.283738:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134509568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697947.283743:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134509696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697947.283760:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697947.283770:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697947/real 1787697947] req@ffff9d85b9d2f800 x1874531134509696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697947.283780:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134509696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697947.283787:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697947.283790:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697947.283793:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134509696:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697947.859039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697948.883012:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697949.906996:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697950.931038:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697951.955063:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697952.403011:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697952.403026:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697952.403031:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697952.403043:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5854 00000100:00080000:3.0:1787697952.403056:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697952.403065:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697952.403113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134509824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697952.403145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697952.403149:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697952.403151:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697952.403156:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5854 00000100:00080000:3.0:1787697952.403160:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697952.403163:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697952.403175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697952.403188:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134509952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697952.403197:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134509824:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697952.403197:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697952.403200:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697952.403202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697952.403205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5854 00000100:00080000:3.0:1787697952.403208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697952.403212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697952.403217:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697952.403225:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134510080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697952.403232:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697952.403235:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697952.403236:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697952.403239:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5854 00000100:00080000:3.0:1787697952.403241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697952.403244:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697952.403248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697952.403257:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134510208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697952.403264:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697952.403266:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697952.403268:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697952.403270:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697952.403271:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5854 00000100:00080000:3.0:1787697952.403274:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697952.403277:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697952.403281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697952.403289:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134510336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697952.403297:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697952.403300:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697952.403301:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697952.403305:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5854 00000100:00080000:3.0:1787697952.403307:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697952.403310:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697952.403314:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697952.403319:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697952/real 1787697952] req@ffff9d85c0fb1c00 x1874531134509824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697952.403322:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134510464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697952.403330:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697952.403333:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697952.403335:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697952.403338:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134509824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697952.403338:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5854 00000100:00080000:3.0:1787697952.403341:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697952.403345:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697952.403349:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697952.403349:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697952.403353:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697952.403357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134509824:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697952.403357:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134510592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697952.403384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697952.403406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697952.403418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697952/real 1787697952] req@ffff9d85c0fb3800 x1874531134509952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.403426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134509952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.403431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.403434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697952.403436:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134509952:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697952.403439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697952.403453:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697952.403459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697952/real 1787697952] req@ffff9d85c0fb0a80 x1874531134510080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.403465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134510080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.403470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.403471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697952.403473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697952.403476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697952.403484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697952.403489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697952/real 1787697952] req@ffff9d85c0fb0e00 x1874531134510208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.403496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134510208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.403501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.403502:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697952.403504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697952.403507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134510336:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697952.403515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697952.403520:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697952/real 1787697952] req@ffff9d85c0fb2680 x1874531134510336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.403526:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134510336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.403530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.403531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697952.403533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134510336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697952.403535:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134510464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697952.403543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697952.403548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697952/real 1787697952] req@ffff9d85c0fb2a00 x1874531134510464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.403553:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134510464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.403562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.403563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697952.403565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134510464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697952.403568:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134510592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697952.403576:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697952.403580:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697952/real 1787697952] req@ffff9d85c0fb3100 x1874531134510592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.403586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134510592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.403590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.403591:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697952.403593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134510592:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697952.979043:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697954.003059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697955.027054:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697956.051946:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697957.075008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697957.522905:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697957.522912:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697957.522914:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697957.522922:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5859 00000100:00080000:3.0:1787697957.522930:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697957.522935:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697957.522966:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134510720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697957.522986:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697957.522988:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697957.522989:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697957.522995:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5859 00000100:00080000:3.0:1787697957.522997:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697957.522999:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697957.523005:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697957.523014:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134510848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697957.523020:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697957.523022:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697957.523023:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697957.523025:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5859 00000100:00080000:3.0:1787697957.523027:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697957.523031:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523032:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134510720:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697957.523034:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697957.523047:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134510976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697957.523052:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697957.523054:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697957.523056:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697957.523058:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5859 00000100:00080000:3.0:1787697957.523060:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697957.523062:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697957.523065:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697957.523072:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134511104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697957.523078:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697957.523080:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697957.523081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697957.523086:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697957.523089:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5859 00000100:00080000:3.0:1787697957.523092:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697957.523096:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697957.523099:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697957.523106:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134511232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697957.523110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697957.523113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697957.523114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697957.523117:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5859 00000100:00080000:3.0:1787697957.523119:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697957.523124:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697957/real 1787697957] req@ffff9d85c0fb3b80 x1874531134510720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697957.523124:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697957.523126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697957.523137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134511360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697957.523144:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134510720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697957.523145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697957.523154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697957.523159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697957.523160:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697957.523165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5859 00000100:00080000:0.0:1787697957.523166:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697957.523172:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134510720:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697957.523173:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697957.523181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523183:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510848:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787697957.523189:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697957.523204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134511488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697957.523205:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523214:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697957/real 1787697957] req@ffff9d85c0fb0380 x1874531134510848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697957.523224:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134510848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697957.523230:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697957.523233:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697957.523235:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510848:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697957.523250:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697957.523264:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697957/real 1787697957] req@ffff9d85c0fb2d80 x1874531134510976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697957.523283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134510976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697957.523291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697957.523294:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697957.523297:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134510976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697957.523303:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134511104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697957.523321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523331:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697957/real 1787697957] req@ffff9d85c0fb3100 x1874531134511104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697957.523343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134511104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697957.523350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697957.523353:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697957.523356:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134511104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697957.523361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134511232:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697957.523378:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523385:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697957/real 1787697957] req@ffff9d85c0fb2a00 x1874531134511232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697957.523392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134511232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697957.523397:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697957.523399:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697957.523400:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134511232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697957.523403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134511360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697957.523412:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523417:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697957/real 1787697957] req@ffff9d85c0fb2680 x1874531134511360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697957.523423:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134511360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697957.523432:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697957.523435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697957.523437:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134511360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697957.523442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134511488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697957.523458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697957.523467:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697957/real 1787697957] req@ffff9d85c0fb0e00 x1874531134511488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697957.523478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134511488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697957.523485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697957.523486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697957.523488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134511488:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697958.099009:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697959.122958:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697960.146994:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697961.170960:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697962.194972:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697962.642970:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697962.642977:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697962.642980:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697962.642988:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5864 00000100:00080000:2.0:1787697962.642996:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697962.643002:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787697962.643032:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134511616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697962.643050:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697962.643052:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697962.643053:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697962.643057:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5864 00000100:00080000:2.0:1787697962.643059:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697962.643061:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697962.643068:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697962.643076:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134511744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697962.643083:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697962.643085:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697962.643086:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697962.643089:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5864 00000100:00100000:0.0:1787697962.643093:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134511616:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787697962.643093:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697962.643097:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697962.643103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697962.643110:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134511872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697962.643115:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697962.643117:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697962.643118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697962.643120:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5864 00000100:00080000:2.0:1787697962.643122:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697962.643124:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697962.643127:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697962.643134:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134512000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697962.643140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697962.643142:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697962.643143:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697962.643144:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5864 00000100:00080000:2.0:1787697962.643147:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697962.643150:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787697962.643153:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697962.643157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697962.643165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134512128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697962.643170:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697962.643172:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697962.643173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697962.643176:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5864 00000100:00080000:2.0:1787697962.643178:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697962.643180:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697962.643183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697962.643187:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697962/real 1787697962] req@ffff9d85babd6d80 x1874531134511616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697962.643191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134512256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697962.643210:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697962.643212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697962.643214:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697962.643218:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134511616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697962.643220:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5864 00000100:00080000:2.0:1787697962.643225:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697962.643231:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787697962.643235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697962.643236:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697962.643242:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787697962.643243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134512384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697962.643247:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134511616:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697962.643272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134511744:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697962.643287:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697962.643294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697962/real 1787697962] req@ffff9d85babd5500 x1874531134511744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697962.643300:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134511744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697962.643305:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697962.643307:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697962.643309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134511744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697962.643312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134511872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697962.643321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697962.643326:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697962/real 1787697962] req@ffff9d85babd7800 x1874531134511872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697962.643332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134511872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697962.643336:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697962.643338:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697962.643340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134511872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697962.643342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697962.643350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697962.643355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697962/real 1787697962] req@ffff9d85babd6680 x1874531134512000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697962.643361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134512000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697962.643366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697962.643367:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697962.643369:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697962.643371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134512128:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697962.643379:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697962.643384:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697962/real 1787697962] req@ffff9d85babd6a00 x1874531134512128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697962.643394:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134512128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697962.643401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697962.643403:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697962.643406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134512128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697962.643412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134512256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697962.643429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697962.643439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697962/real 1787697962] req@ffff9d85babd6300 x1874531134512256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697962.643446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134512256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697962.643455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697962.643457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697962.643458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134512256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697962.643462:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134512384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697962.643473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697962.643478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697962/real 1787697962] req@ffff9d85babd4380 x1874531134512384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697962.643485:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134512384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697962.643489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697962.643491:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697962.643492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134512384:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697963.219012:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697964.243028:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697965.266998:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697966.290995:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697967.314984:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697967.762956:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697967.762964:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697967.762967:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697967.762975:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5869 00000100:00080000:1.0:1787697967.762984:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697967.762990:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697967.763018:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134512512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697967.763041:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697967.763044:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697967.763045:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697967.763048:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5869 00000100:00080000:1.0:1787697967.763051:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697967.763053:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697967.763061:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697967.763071:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134512640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697967.763079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697967.763081:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697967.763082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697967.763087:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134512512:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697967.763088:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5869 00000100:00080000:1.0:1787697967.763092:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697967.763096:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697967.763099:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697967.763106:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134512768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697967.763112:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697967.763115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697967.763116:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697967.763118:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5869 00000100:00080000:1.0:1787697967.763120:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697967.763122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697967.763125:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697967.763131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134512896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697967.763137:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697967.763139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697967.763140:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697967.763143:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5869 00000800:00000100:0.0:1787697967.763146:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697967.763147:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697967.763152:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697967.763157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697967.763168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134513024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697967.763177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697967.763181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697967.763182:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697967/real 1787697967] req@ffff9d85b9d2dc00 x1874531134512512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697967.763184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697967.763194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5869 00000100:00080000:1.0:1787697967.763199:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697967.763203:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134512512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697967.763206:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697967.763212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697967.763223:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697967.763227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134513152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697967.763232:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697967.763239:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697967.763240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134512512:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697967.763243:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697967.763247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697967.763251:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5869 00000100:00080000:1.0:1787697967.763253:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697967.763256:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697967.763258:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697967.763265:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134513280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697967.763269:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512640:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697967.763294:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697967.763303:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697967/real 1787697967] req@ffff9d85b9d2c380 x1874531134512640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697967.763311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134512640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697967.763316:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697967.763318:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697967.763320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697967.763324:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697967.763334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697967.763339:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697967/real 1787697967] req@ffff9d85b9d2c700 x1874531134512768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697967.763346:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134512768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697967.763350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697967.763352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697967.763353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697967.763357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697967.763366:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697967.763371:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697967/real 1787697967] req@ffff9d85b9d2ed80 x1874531134512896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697967.763378:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134512896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697967.763382:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697967.763384:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697967.763385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134512896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697967.763388:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513024:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697967.763396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697967.763401:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697967/real 1787697967] req@ffff9d85b9d2d500 x1874531134513024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697967.763407:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134513024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697967.763411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697967.763413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697967.763416:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697967.763420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697967.763437:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697967.763447:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697967/real 1787697967] req@ffff9d85b9d2e680 x1874531134513152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697967.763458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134513152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697967.763471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697967.763473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697967.763476:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697967.763491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697967.763509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697967.763518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697967/real 1787697967] req@ffff9d85b9d2dc00 x1874531134513280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697967.763541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134513280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697967.763549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697967.763552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697967.763555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513280:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697968.339048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697969.363001:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697970.387029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697971.411068:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697972.434996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697972.882977:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697972.882984:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697972.882988:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697972.882996:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5874 00000100:00080000:3.0:1787697972.883006:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697972.883012:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697972.883046:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134513408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697972.883066:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697972.883068:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697972.883069:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697972.883072:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5874 00000100:00080000:3.0:1787697972.883074:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697972.883077:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697972.883083:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697972.883094:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134513536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697972.883102:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134513408:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697972.883106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697972.883112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697972.883114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697972.883119:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5874 00000100:00080000:3.0:1787697972.883123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697972.883126:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697972.883131:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697972.883143:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134513664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697972.883152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697972.883155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697972.883157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697972.883161:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5874 00000800:00000100:0.0:1787697972.883164:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697972.883167:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697972.883172:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697972.883177:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697972.883186:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134513792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697972.883192:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697972.883195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697972.883196:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697972.883200:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697972/real 1787697972] req@ffff9d85c0fb0a80 x1874531134513408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697972.883200:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5874 00000100:00080000:3.0:1787697972.883206:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697972.883212:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697972.883216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697972.883220:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134513408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697972.883225:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134513920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697972.883236:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697972.883238:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697972.883242:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787697972.883242:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697972.883244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697972.883247:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134513408:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697972.883249:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5874 00000100:00080000:3.0:1787697972.883255:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697972.883258:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697972.883260:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697972.883266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134514048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697972.883272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134513536:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697972.883273:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697972.883278:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697972.883279:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697972.883282:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5874 00000100:00080000:3.0:1787697972.883285:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697972.883287:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697972.883290:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697972.883294:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697972.883302:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134514176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697972.883306:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697972/real 1787697972] req@ffff9d85c0fb3800 x1874531134513536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697972.883320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134513536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697972.883325:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697972.883327:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697972.883329:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134513536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697972.883333:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134513664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697972.883345:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697972.883350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697972/real 1787697972] req@ffff9d85c0fb1c00 x1874531134513664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697972.883356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134513664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697972.883361:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697972.883362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697972.883364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134513664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697972.883367:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134513792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697972.883374:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697972.883379:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697972/real 1787697972] req@ffff9d85c0fb3480 x1874531134513792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697972.883386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134513792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697972.883390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697972.883391:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697972.883393:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134513792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697972.883396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513920:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697972.883410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697972.883420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697972/real 1787697972] req@ffff9d85c0fb2300 x1874531134513920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697972.883431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134513920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697972.883438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697972.883466:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697972.883468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134513920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697972.883480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697972.883494:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697972.883500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697972/real 1787697972] req@ffff9d85c0fb1880 x1874531134514048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697972.883507:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134514048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697972.883515:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697972.883517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697972.883519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697972.883522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697972.883530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697972.883534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697972/real 1787697972] req@ffff9d85c0fb1500 x1874531134514176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697972.883543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134514176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697972.883550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697972.883552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697972.883555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514176:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697973.459010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697974.482997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697975.507022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697976.531041:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697977.555021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697978.002978:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697978.002988:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697978.002993:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697978.003003:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5879 00000100:00080000:3.0:1787697978.003014:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697978.003022:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697978.003060:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134514304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697978.003089:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697978.003092:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697978.003094:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697978.003098:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5879 00000100:00080000:3.0:1787697978.003102:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697978.003106:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697978.003115:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697978.003128:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134514432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697978.003137:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697978.003141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697978.003143:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697978.003146:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5879 00000100:00080000:3.0:1787697978.003149:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697978.003154:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697978.003159:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134514304:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787697978.003166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697978.003180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134514560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697978.003197:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697978.003200:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697978.003201:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697978.003204:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5879 00000100:00080000:3.0:1787697978.003208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697978.003213:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697978.003218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697978.003232:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134514688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697978.003241:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697978.003245:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697978.003247:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697978.003251:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5879 00000100:00080000:3.0:1787697978.003255:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697978.003262:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697978.003263:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134514432:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787697978.003269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697978.003283:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134514816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697978.003291:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697978.003294:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697978.003295:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697978.003298:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5879 00000100:00080000:3.0:1787697978.003302:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697978.003306:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697978.003310:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697978.003312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134514560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787697978.003318:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134514944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697978.003331:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697978.003334:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697978.003335:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697978.003338:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5879 00000100:00100000:0.0:1787697978.003342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134514688:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787697978.003343:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697978.003351:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697978.003370:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697978.003384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787697978.003386:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134515072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697978.003420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697978.003444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515072:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697978.578959:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697979.603008:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697980.627025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697981.075115:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787697981.075155:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697981.075171:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697981.075218:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697981.075226:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697981.075234:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697981.075240:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697981.075247:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787697981.075254:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787697981.075356:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697978/real 1787697981] req@ffff9d85c0fb2300 x1874531134514944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697981.075379:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134514944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697981.075390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697981.075395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697981.075399:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697981.075406:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697978/real 1787697981] req@ffff9d85c0fb3480 x1874531134515072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697981.075417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134515072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697981.075425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697981.075428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697981.075431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697981.075437:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697978/real 1787697981] req@ffff9d85c0fb0700 x1874531134514560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697981.075447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134514560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697981.075455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697981.075458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697981.075461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134514560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697981.075468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697978/real 1787697981] req@ffff9d85c0fb1500 x1874531134514688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697981.075477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134514688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697981.075484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697981.075487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697981.075490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134514688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697981.075496:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697978/real 1787697981] req@ffff9d85c0fb1880 x1874531134514816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697981.075505:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134514816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697981.075512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697981.075515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697981.075518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134514816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697981.075524:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697978/real 1787697981] req@ffff9d85c0fb1f80 x1874531134514304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697981.075535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134514304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697981.075541:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697981.075544:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697981.075548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134514304:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787697981.075554:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697978/real 1787697981] req@ffff9d85c0fb0000 x1874531134514432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697981.075564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134514432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697981.075571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697981.075574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697981.075577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134514432:192.168.201.137@tcp:38:kworker.0 00010000:00010000:3.0:1787697981.651006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697982.674987:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697983.122966:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697983.122972:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697983.122976:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697983.122984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5884 00000100:00080000:1.0:1787697983.122992:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697983.122997:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697983.123023:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134515200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697983.123043:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697983.123045:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697983.123047:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697983.123049:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5884 00000100:00080000:1.0:1787697983.123051:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697983.123054:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697983.123060:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697983.123069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134515328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697983.123076:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697983.123079:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697983.123081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697983.123084:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5884 00000100:00080000:1.0:1787697983.123087:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697983.123091:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697983.123097:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697983.123098:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134515200:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787697983.123111:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134515456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697983.123121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697983.123124:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697983.123127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697983.123130:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5884 00000100:00080000:1.0:1787697983.123134:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697983.123138:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697983.123142:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697983.123151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134515584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697983.123161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697983.123165:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787697983.123167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787697983.123169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697983.123175:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5884 00000100:00080000:1.0:1787697983.123180:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697983.123184:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697983.123189:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697983.123202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134515712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697983.123208:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697983/real 1787697983] req@ffff9d85b9d2f800 x1874531134515200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697983.123213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697983.123221:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697983.123224:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697983.123230:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5884 00000100:00080000:1.0:1787697983.123234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697983.123238:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134515200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697983.123241:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697983.123246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697983.123260:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697983.123260:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134515840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697983.123281:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697983.123289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134515200:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697983.123290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697983.123298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697983.123300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697983.123301:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134515328:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787697983.123309:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5884 00000100:00080000:1.0:1787697983.123313:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697983.123317:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697983.123323:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697983.123333:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697983.123338:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134515968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697983.123348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697983/real 1787697983] req@ffff9d85b9d2fb80 x1874531134515328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697983.123370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134515328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697983.123379:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697983.123382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697983.123386:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134515328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697983.123408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134515456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697983.123428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697983.123439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697983/real 1787697983] req@ffff9d85b9d2e300 x1874531134515456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697983.123451:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134515456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697983.123459:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697983.123462:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697983.123464:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134515456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697983.123470:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134515584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697983.123489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697983.123500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697983/real 1787697983] req@ffff9d85b9d2ce00 x1874531134515584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697983.123511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134515584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697983.123523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697983.123526:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697983.123529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134515584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697983.123535:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515712:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697983.123551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697983.123559:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697983/real 1787697983] req@ffff9d85b9d2c000 x1874531134515712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697983.123569:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134515712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697983.123576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697983.123579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697983.123582:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697983.123587:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697983.123602:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697983.123610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697983/real 1787697983] req@ffff9d85b9d2ca80 x1874531134515840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697983.123621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134515840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697983.123628:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697983.123630:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697983.123633:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697983.123638:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697983.123655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697983.123664:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697983/real 1787697983] req@ffff9d85b9d2f480 x1874531134515968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697983.123675:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134515968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697983.123683:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697983.123686:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697983.123689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134515968:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697983.699026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697984.722986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697985.747030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697986.771017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697987.794999:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697988.242945:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697988.242951:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697988.242954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697988.242961:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5890 00000100:00080000:1.0:1787697988.242975:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697988.242983:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787697988.243017:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134516096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697988.243043:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697988.243045:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697988.243046:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697988.243049:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5890 00000100:00080000:1.0:1787697988.243052:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697988.243054:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697988.243061:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697988.243070:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134516224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697988.243076:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697988.243078:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697988.243079:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697988.243082:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5890 00000100:00080000:1.0:1787697988.243083:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697988.243085:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697988.243089:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697988.243095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134516352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697988.243099:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134516096:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697988.243103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697988.243108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697988.243109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697988.243112:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5890 00000100:00080000:1.0:1787697988.243114:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697988.243115:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697988.243118:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697988.243126:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134516480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697988.243131:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697988.243133:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697988.243134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697988.243136:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5890 00000100:00080000:1.0:1787697988.243151:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697988.243154:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697988.243156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697988.243162:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787697988.243167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134516608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697988.243202:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697988.243206:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697988.243208:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697988.243214:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5890 00000100:00100000:0.0:1787697988.243216:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697988/real 1787697988] req@ffff9d85b9d2f480 x1874531134516096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697988.243218:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697988.243224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697988.243230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697988.243243:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134516096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697988.243243:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134516736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697988.243274:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697988.243274:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697988.243279:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697988.243281:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697988.243283:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787697988.243286:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5890 00000100:00080000:1.0:1787697988.243290:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697988.243292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134516096:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787697988.243294:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787697988.243302:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697988.243307:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134516224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787697988.243313:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134516864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697988.243331:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697988.243344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697988/real 1787697988] req@ffff9d85b9d2ca80 x1874531134516224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697988.243355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134516224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697988.243362:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697988.243365:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697988.243368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134516224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697988.243387:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134516352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697988.243403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697988.243413:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697988/real 1787697988] req@ffff9d85b9d2c000 x1874531134516352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697988.243424:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134516352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697988.243431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697988.243434:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697988.243437:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134516352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697988.243443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134516480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697988.243459:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697988.243468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697988/real 1787697988] req@ffff9d85b9d2ce00 x1874531134516480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697988.243477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134516480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697988.243484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697988.243487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697988.243490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134516480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697988.243495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134516608:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697988.243509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697988.243517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697988/real 1787697988] req@ffff9d85b9d2e300 x1874531134516608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697988.243526:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134516608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697988.243533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697988.243536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697988.243539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134516608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697988.243544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134516736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697988.243567:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697988.243577:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697988/real 1787697988] req@ffff9d85b9d2fb80 x1874531134516736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697988.243587:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134516736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697988.243593:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697988.243596:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697988.243599:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134516736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697988.243604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134516864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697988.243618:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697988.243626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697988/real 1787697988] req@ffff9d85b9d2f800 x1874531134516864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697988.243636:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134516864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697988.243643:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697988.243645:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697988.243648:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134516864:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697988.819065:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697989.843007:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697990.869279:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697991.890983:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697992.915016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697993.362986:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697993.362996:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697993.363000:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697993.363008:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5895 00000100:00080000:3.0:1787697993.363019:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697993.363026:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697993.363063:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134516992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697993.363102:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697993.363105:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697993.363107:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697993.363111:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5895 00000100:00080000:3.0:1787697993.363115:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697993.363119:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697993.363129:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697993.363142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134517120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697993.363147:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134516992:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697993.363158:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697993.363162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697993.363163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697993.363168:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5895 00000100:00080000:3.0:1787697993.363173:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697993.363176:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697993.363182:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697993.363194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134517248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697993.363203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697993.363206:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697993.363208:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697993.363213:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787697993.363216:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5895 00000100:00080000:3.0:1787697993.363223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697993.363227:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697993.363232:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697993.363245:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134517376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697993.363255:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697993.363256:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697993/real 1787697993] req@ffff9d85c0fb1c00 x1874531134516992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697993.363262:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697993.363265:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697993.363272:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5895 00000100:00080000:3.0:1787697993.363278:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697993.363282:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134516992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697993.363286:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697993.363294:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697993.363298:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697993.363304:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787697993.363308:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134517504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697993.363312:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134516992:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697993.363328:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697993.363332:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697993.363333:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697993.363338:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5895 00000100:00100000:0.0:1787697993.363341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134517120:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697993.363343:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697993.363351:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697993.363356:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697993.363368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787697993.363371:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134517632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697993.363383:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697993/real 1787697993] req@ffff9d85c0fb3800 x1874531134517120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697993.363400:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134517120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697993.363402:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697993.363412:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697993.363415:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697993.363418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697993.363421:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5895 00000100:00080000:0.0:1787697993.363425:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697993.363431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134517120:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787697993.363432:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697993.363440:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787697993.363443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134517248:192.168.201.137@tcp:8:kworker.0 00000008:00100000:3.0:1787697993.363447:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697993.363462:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134517760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697993.363469:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697993.363481:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697993/real 1787697993] req@ffff9d85c0fb0a80 x1874531134517248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697993.363495:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134517248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697993.363503:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697993.363507:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697993.363511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134517248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697993.363517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134517376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697993.363532:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697993.363540:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697993/real 1787697993] req@ffff9d85c0fb0e00 x1874531134517376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697993.363551:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134517376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697993.363568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697993.363572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697993.363578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134517376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697993.363583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134517504:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697993.363601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697993.363612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697993/real 1787697993] req@ffff9d85c0fb2680 x1874531134517504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697993.363623:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134517504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697993.363630:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697993.363634:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697993.363637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134517504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697993.363652:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134517632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697993.363671:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697993.363680:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697993/real 1787697993] req@ffff9d85c0fb2a00 x1874531134517632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697993.363691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134517632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697993.363699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697993.363702:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697993.363705:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134517632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697993.363710:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134517760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697993.363727:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697993.363737:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697993/real 1787697993] req@ffff9d85c0fb3100 x1874531134517760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697993.363747:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134517760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697993.363755:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697993.363758:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697993.363760:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134517760:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787697993.939005:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697994.962999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697995.986977:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697997.011044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787697998.035069:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697998.482934:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697998.482945:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697998.482949:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697998.482959:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5900 00000100:00080000:3.0:1787697998.482971:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697998.482978:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787697998.483040:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134517888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697998.483068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697998.483071:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697998.483073:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697998.483078:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5900 00000100:00080000:3.0:1787697998.483083:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697998.483087:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697998.483096:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697998.483109:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134518016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697998.483111:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134517888:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787697998.483121:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697998.483124:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697998.483126:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697998.483129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5900 00000100:00080000:3.0:1787697998.483131:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697998.483133:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697998.483137:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697998.483144:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134518144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697998.483149:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697998.483152:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697998.483153:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697998.483156:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5900 00000100:00080000:3.0:1787697998.483158:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697998.483160:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697998.483163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697998.483171:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134518272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697998.483181:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697998.483184:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697998.483185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697998.483187:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5900 00000100:00080000:3.0:1787697998.483189:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697998.483194:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787697998.483195:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787697998.483197:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697998.483206:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134518400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697998.483212:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697998.483219:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697998.483220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697998.483223:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5900 00000100:00080000:3.0:1787697998.483225:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697998.483228:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787697998.483232:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697998.483243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134518528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697998.483248:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697998/real 1787697998] req@ffff9d85c0fb2d80 x1874531134517888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697998.483259:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697998.483265:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697998.483267:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697998.483273:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5900 00000100:00100000:0.0:1787697998.483277:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134517888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697998.483279:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697998.483288:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787697998.483295:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697998.483296:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697998.483303:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697998.483309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134517888:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787697998.483314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134518656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697998.483341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697998.483363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697998.483375:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697998/real 1787697998] req@ffff9d85c0fb0380 x1874531134518016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697998.483403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134518016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697998.483412:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697998.483415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697998.483419:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697998.483424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697998.483443:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697998.483453:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697998/real 1787697998] req@ffff9d85c0fb3b80 x1874531134518144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697998.483463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134518144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697998.483471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697998.483474:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697998.483477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697998.483483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697998.483499:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697998.483508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697998/real 1787697998] req@ffff9d85c0fb3100 x1874531134518272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697998.483518:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134518272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697998.483526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697998.483529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697998.483532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697998.483537:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134518400:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787697998.483552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697998.483567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697998/real 1787697998] req@ffff9d85c0fb2a00 x1874531134518400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697998.483581:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134518400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697998.483588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697998.483592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697998.483595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134518400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787697998.483600:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134518528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697998.483616:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697998.483625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697998/real 1787697998] req@ffff9d85c0fb2680 x1874531134518528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697998.483635:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134518528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697998.483643:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697998.483646:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697998.483649:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134518528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787697998.483654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134518656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787697998.483669:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787697998.483677:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697998/real 1787697998] req@ffff9d85c0fb0e00 x1874531134518656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697998.483687:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134518656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697998.483694:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697998.483697:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787697998.483700:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134518656:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787697999.059008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698000.083022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698001.106990:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698002.130997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698003.155010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698003.603059:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698003.603078:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698003.603081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698003.603124:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5905 00000100:00080000:3.0:1787698003.603168:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698003.603185:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698003.603217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134518784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698003.603281:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698003.603286:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698003.603288:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698003.603293:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5905 00000100:00080000:3.0:1787698003.603297:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698003.603300:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698003.603309:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698003.603311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134518784:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698003.603351:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134518912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698003.603367:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698003.603378:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698003.603382:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698003.603383:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698003.603389:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5905 00000100:00080000:3.0:1787698003.603393:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698003.603397:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698003.603403:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698003.603410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698003/real 1787698003] req@ffff9d85c0fb0a80 x1874531134518784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698003.603417:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134519040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698003.603427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134518784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698003.603429:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698003.603441:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698003.603443:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698003.603444:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698003.603451:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698003.603451:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5905 00000100:00080000:3.0:1787698003.603459:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698003.603460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134518784:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698003.603467:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698003.603472:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698003.603484:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134519168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698003.603488:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518912:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698003.603501:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698003.603504:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698003.603507:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698003.603511:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5905 00000800:00000100:0.0:1787698003.603516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698003.603518:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698003.603524:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698003.603528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698003/real 1787698003] req@ffff9d85c0fb3800 x1874531134518912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698003.603538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134518912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698003.603543:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698003.603546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787698003.603548:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698003.603549:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134518912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698003.603557:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698003.603569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698003.603575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698003/real 1787698003] req@ffff9d85c0fb1c00 x1874531134519040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698003.603585:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134519040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698003.603590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698003.603592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698003.603595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698003.603595:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134519296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698003.603605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519168:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698003.603619:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698003.603623:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698003.603625:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698003.603635:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698003/real 1787698003] req@ffff9d85c0fb0000 x1874531134519168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698003.603639:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698003.603646:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5905 00000100:00080000:3.0:1787698003.603653:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698003.603657:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134519168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698003.603666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698003.603668:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698003.603671:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698003.603672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698003.603685:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134519296:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787698003.603694:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698003.603701:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698003.603711:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698003/real 1787698003] req@ffff9d85c0fb1f80 x1874531134519296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698003.603711:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134519424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698003.603719:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134519296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698003.603724:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698003.603725:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698003.603729:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134519296:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698003.603750:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698003.603754:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698003.603756:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698003.603769:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134519424:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698003.603775:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5905 00000100:00080000:3.0:1787698003.603781:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698003.603786:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698003.603795:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698003.603799:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698003/real 1787698003] req@ffff9d85c0fb1880 x1874531134519424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698003.603802:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698003.603816:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134519424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698003.603825:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698003.603827:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134519552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698003.603831:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698003.603838:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134519424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698003.603952:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134519552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698003.603973:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698003.603982:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698003/real 1787698003] req@ffff9d85c0fb1500 x1874531134519552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698003.603993:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134519552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698003.604001:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698003.604005:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698003.604008:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134519552:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698004.179029:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698005.203015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698006.227025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698007.251030:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698008.274986:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698008.722937:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698008.722946:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698008.722951:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698008.722960:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5910 00000100:00080000:2.0:1787698008.722969:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698008.722976:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698008.723009:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134519680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698008.723035:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698008.723039:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698008.723041:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698008.723045:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5910 00000100:00080000:2.0:1787698008.723049:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698008.723054:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698008.723062:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698008.723066:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134519680:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698008.723078:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134519808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698008.723089:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698008.723092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698008.723094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698008.723098:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5910 00000100:00080000:2.0:1787698008.723102:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698008.723106:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698008.723111:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698008.723120:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698008.723125:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134519936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698008.723135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698008.723139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698008.723141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698008.723145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5910 00000100:00080000:2.0:1787698008.723148:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698008.723152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698008.723158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698008.723162:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698008/real 1787698008] req@ffff9d85babd4000 x1874531134519680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698008.723172:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134520064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698008.723176:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134519680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698008.723198:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698008.723199:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698008.723202:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698008.723207:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787698008.723210:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134519680:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698008.723211:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698008.723218:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5910 00000100:00080000:2.0:1787698008.723223:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698008.723227:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698008.723232:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698008.723238:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698008.723245:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134520192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698008.723256:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698008.723260:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698008.723263:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698008.723264:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698008.723272:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5910 00000100:00080000:2.0:1787698008.723277:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698008.723278:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698008/real 1787698008] req@ffff9d85babd5c00 x1874531134519808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698008.723284:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698008.723291:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134519808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698008.723292:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698008.723302:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698008.723304:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698008.723310:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698008.723311:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134520320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698008.723315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519936:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698008.723329:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698008.723333:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698008.723334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698008.723336:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698008.723342:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5910 00000100:00100000:0.0:1787698008.723344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698008/real 1787698008] req@ffff9d85babd7480 x1874531134519936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698008.723349:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698008.723355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134519936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698008.723357:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698008.723365:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698008.723367:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698008.723369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698008.723374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134519936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698008.723378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698008.723385:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134520448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698008.723392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698008.723400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698008/real 1787698008] req@ffff9d85babd5180 x1874531134520064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698008.723408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134520064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698008.723419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698008.723421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698008.723423:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698008.723432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134520192:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698008.723442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698008.723448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698008/real 1787698008] req@ffff9d85babd4e00 x1874531134520192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698008.723454:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134520192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698008.723458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698008.723459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698008.723461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134520192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698008.723464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134520320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698008.723472:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698008.723479:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698008/real 1787698008] req@ffff9d85babd7b80 x1874531134520320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698008.723489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134520320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698008.723496:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698008.723499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698008.723502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134520320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698008.723507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134520448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698008.723522:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698008.723531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698008/real 1787698008] req@ffff9d85babd7100 x1874531134520448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698008.723540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134520448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698008.723547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698008.723550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698008.723552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134520448:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698009.299010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698010.322982:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698011.347014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698012.371004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698013.395011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698013.842944:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698013.842951:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698013.842954:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698013.842961:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5915 00000100:00080000:3.0:1787698013.842969:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698013.842974:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698013.843010:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134520576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698013.843026:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698013.843029:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698013.843030:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698013.843033:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5915 00000100:00080000:3.0:1787698013.843035:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698013.843037:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698013.843043:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698013.843051:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134520704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698013.843057:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698013.843059:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698013.843060:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698013.843066:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134520576:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698013.843067:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5915 00000100:00080000:3.0:1787698013.843072:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698013.843077:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698013.843082:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698013.843095:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134520832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698013.843104:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698013.843108:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698013.843110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698013.843113:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5915 00000100:00080000:3.0:1787698013.843115:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698013.843118:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698013.843121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698013.843133:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698013.843133:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134520960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698013.843142:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698013.843145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698013.843153:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698013.843158:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5915 00000100:00080000:3.0:1787698013.843160:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698013.843163:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698013.843167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698013.843173:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698013/real 1787698013] req@ffff9d85c0fb0700 x1874531134520576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698013.843175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134521088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698013.843198:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698013.843201:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134520576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698013.843203:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698013.843207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698013.843214:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5915 00000100:00080000:0.0:1787698013.843216:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698013.843217:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698013.843221:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698013.843223:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698013.843228:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134520576:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698013.843230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698013.843240:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134521216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698013.843247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520704:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698013.843249:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698013.843253:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698013.843255:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698013.843260:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5915 00000100:00080000:3.0:1787698013.843265:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698013.843267:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698013.843272:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698013.843278:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698013/real 1787698013] req@ffff9d85c0fb3480 x1874531134520704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698013.843279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698013.843292:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134520704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698013.843296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134521344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698013.843302:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698013.843309:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698013.843313:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698013.843316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698013.843328:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698013.843334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698013/real 1787698013] req@ffff9d85c0fb2300 x1874531134520832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698013.843341:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134520832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698013.843345:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698013.843346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698013.843348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698013.843351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698013.843359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698013.843364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698013/real 1787698013] req@ffff9d85c0fb1500 x1874531134520960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698013.843370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134520960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698013.843485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698013.843488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698013.843491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134520960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698013.843494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698013.843509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698013.843517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698013/real 1787698013] req@ffff9d85c0fb1880 x1874531134521088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698013.843523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134521088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698013.843528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698013.843530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698013.843531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698013.843541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698013.843550:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698013.843555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698013/real 1787698013] req@ffff9d85c0fb1f80 x1874531134521216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698013.843561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134521216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698013.843566:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698013.843567:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698013.843569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698013.843572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698013.843581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698013.843587:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698013/real 1787698013] req@ffff9d85c0fb0000 x1874531134521344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698013.843597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134521344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698013.843604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698013.843607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698013.843610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521344:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698014.419013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698015.442993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698016.467022:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698017.491000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698018.514998:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698018.962958:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698018.962965:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698018.962969:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698018.962976:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5920 00000100:00080000:2.0:1787698018.962985:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698018.962991:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698018.963019:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134521472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698018.963039:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698018.963041:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698018.963042:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698018.963045:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5920 00000100:00080000:2.0:1787698018.963047:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698018.963049:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698018.963055:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698018.963064:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134521600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698018.963071:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698018.963078:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698018.963080:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698018.963084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134521472:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698018.963086:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5920 00000100:00080000:2.0:1787698018.963092:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698018.963094:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698018.963098:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698018.963105:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134521728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698018.963111:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698018.963113:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698018.963114:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698018.963117:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5920 00000100:00080000:2.0:1787698018.963119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698018.963120:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698018.963123:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698018.963136:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134521856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698018.963143:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698018.963146:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698018.963149:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698018.963153:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698018.963158:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5920 00000100:00080000:2.0:1787698018.963161:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698018.963166:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698018.963171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698018.963184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134521984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698018.963191:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698018.963193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698018.963195:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698018.963198:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698018/real 1787698018] req@ffff9d85babd5f80 x1874531134521472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698018.963199:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5920 00000100:00080000:2.0:1787698018.963204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698018.963209:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698018.963216:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698018.963226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134522112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698018.963230:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134521472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698018.963245:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698018.963250:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698018.963250:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698018.963255:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698018.963259:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698018.963262:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5920 00000100:00080000:2.0:1787698018.963266:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698018.963274:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698018.963275:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134521472:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698018.963284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134521600:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787698018.963285:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698018.963300:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134522240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698018.963314:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698018.963337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698018/real 1787698018] req@ffff9d85babd4700 x1874531134521600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698018.963348:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134521600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698018.963360:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698018.963362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698018.963364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134521600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698018.963381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134521728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698018.963394:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698018.963401:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698018/real 1787698018] req@ffff9d85babd4a80 x1874531134521728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698018.963407:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134521728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698018.963413:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698018.963422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698018.963426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134521728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698018.963435:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134521856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698018.963452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698018.963464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698018/real 1787698018] req@ffff9d85babd7100 x1874531134521856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698018.963474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134521856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698018.963482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698018.963485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698018.963488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134521856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698018.963494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521984:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698018.963512:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698018.963522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698018/real 1787698018] req@ffff9d85babd7b80 x1874531134521984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698018.963533:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134521984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698018.963540:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698018.963543:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698018.963546:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134521984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698018.963552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134522112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698018.963569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698018.963579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698018/real 1787698018] req@ffff9d85babd4e00 x1874531134522112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698018.963591:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134522112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698018.963598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698018.963601:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698018.963604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134522112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698018.963609:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134522240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698018.963624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698018.963633:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698018/real 1787698018] req@ffff9d85babd5180 x1874531134522240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698018.963643:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134522240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698018.963651:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698018.963670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698018.963673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134522240:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698019.539006:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698020.563016:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698021.586993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698022.611006:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698023.634999:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698024.082959:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698024.082970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698024.082974:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698024.082985:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5925 00000100:00080000:0.0:1787698024.082995:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698024.083000:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698024.083030:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134522368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698024.083051:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698024.083053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698024.083054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698024.083057:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5925 00000100:00080000:0.0:1787698024.083059:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698024.083061:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698024.083068:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698024.083077:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134522496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698024.083086:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698024.083089:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698024.083091:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698024.083095:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5925 00000100:00080000:0.0:1787698024.083098:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698024.083103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134522368:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698024.083104:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698024.083113:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698024.083126:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134522624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698024.083133:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698024.083135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698024.083136:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698024.083138:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5925 00000100:00080000:0.0:1787698024.083141:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698024.083143:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698024.083147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698024.083152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134522752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698024.083158:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698024.083161:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698024.083162:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698024.083164:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5925 00000100:00080000:0.0:1787698024.083166:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698024.083171:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698024.083171:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698024.083177:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698024.083183:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134522880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698024.083189:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698024.083191:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698024.083192:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698024.083195:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5925 00000100:00080000:0.0:1787698024.083196:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698024.083198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698024.083200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698024.083206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134523008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698024.083211:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698024.083213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698024.083215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698024.083219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5925 00000100:00100000:1.0:1787698024.083220:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698024/real 1787698024] req@ffff9d85b9d08e00 x1874531134522368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698024.083221:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698024.083226:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698024.083230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698024.083239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134523136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698024.083249:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134522368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698024.083261:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698024.083265:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698024.083269:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134522368:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698024.083276:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134522496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698024.083298:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698024.083310:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698024/real 1787698024] req@ffff9d85b9d08a80 x1874531134522496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698024.083323:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134522496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698024.083332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698024.083335:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698024.083338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134522496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698024.083359:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134522624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698024.083377:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698024.083388:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698024/real 1787698024] req@ffff9d85b9d08000 x1874531134522624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698024.083399:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134522624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698024.083407:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698024.083420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698024.083427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134522624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698024.083433:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134522752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698024.083452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698024.083463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698024/real 1787698024] req@ffff9d85b9d0ad80 x1874531134522752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698024.083474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134522752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698024.083487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698024.083490:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698024.083493:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134522752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698024.083498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134522880:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698024.083515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698024.083524:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698024/real 1787698024] req@ffff9d85b9d0bb80 x1874531134522880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698024.083534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134522880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698024.083542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698024.083545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698024.083548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134522880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698024.083553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698024.083580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698024.083590:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698024/real 1787698024] req@ffff9d85b9d0a680 x1874531134523008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698024.083600:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134523008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698024.083608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698024.083611:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698024.083614:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698024.083619:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698024.083646:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698024.083656:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698024/real 1787698024] req@ffff9d85b9d0b480 x1874531134523136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698024.083667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134523136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698024.083674:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698024.083677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698024.083680:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523136:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698024.659015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698025.683002:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698026.706992:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698027.731010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698028.755018:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698029.202964:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698029.202972:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698029.202975:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698029.202983:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5930 00000100:00080000:3.0:1787698029.202991:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698029.202996:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698029.203025:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134523264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698029.203048:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698029.203050:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698029.203051:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698029.203054:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5930 00000100:00080000:3.0:1787698029.203056:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698029.203058:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698029.203065:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698029.203072:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134523392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698029.203078:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698029.203080:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698029.203081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698029.203084:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5930 00000100:00080000:3.0:1787698029.203085:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698029.203087:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698029.203092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698029.203101:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134523520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698029.203106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134523264:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698029.203118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698029.203121:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698029.203123:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698029.203128:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5930 00000100:00080000:3.0:1787698029.203131:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698029.203135:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698029.203140:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698029.203150:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134523648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698029.203156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698029.203158:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698029.203159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698029.203162:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5930 00000100:00080000:3.0:1787698029.203165:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698029.203166:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698029.203170:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698029.203174:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698029.203181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134523776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698029.203186:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698029.203189:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698029.203190:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698029.203193:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5930 00000100:00080000:3.0:1787698029.203195:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698029.203197:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698029.203200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698029.203210:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698029/real 1787698029] req@ffff9d85c0fb1c00 x1874531134523264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698029.203210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134523904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698029.203235:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698029.203238:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698029.203239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698029.203242:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5930 00000100:00100000:1.0:1787698029.203245:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134523264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698029.203245:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698029.203250:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698029.203256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698029.203262:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698029.203265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134524032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698029.203268:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698029.203274:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134523264:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698029.203283:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134523392:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698029.203300:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698029.203308:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698029/real 1787698029] req@ffff9d85c0fb3800 x1874531134523392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698029.203316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134523392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698029.203328:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698029.203330:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698029.203332:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134523392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698029.203348:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134523520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698029.203359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698029.203365:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698029/real 1787698029] req@ffff9d85c0fb0a80 x1874531134523520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698029.203372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134523520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698029.203376:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698029.203378:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698029.203380:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134523520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698029.203383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134523648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698029.203391:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698029.203396:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698029/real 1787698029] req@ffff9d85c0fb0e00 x1874531134523648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698029.203402:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134523648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698029.203407:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698029.203409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698029.203410:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134523648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698029.203413:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523776:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698029.203421:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698029.203426:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698029/real 1787698029] req@ffff9d85c0fb2680 x1874531134523776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698029.203432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134523776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698029.203436:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698029.203438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698029.203439:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698029.203442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698029.203450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698029.203454:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698029/real 1787698029] req@ffff9d85c0fb2a00 x1874531134523904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698029.203460:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134523904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698029.203464:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698029.203465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698029.203467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134523904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698029.203470:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698029.203505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698029.203517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698029/real 1787698029] req@ffff9d85c0fb3100 x1874531134524032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698029.203528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134524032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698029.203534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698029.203536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698029.203537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524032:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698029.778979:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698030.803010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698031.826990:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698032.851011:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698033.875011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698034.322972:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698034.322981:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698034.322986:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698034.322996:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5936 00000100:00080000:3.0:1787698034.323007:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698034.323014:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698034.323063:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134524160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698034.323099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698034.323102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698034.323103:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698034.323107:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5936 00000100:00080000:3.0:1787698034.323110:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698034.323113:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698034.323122:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698034.323132:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134524288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698034.323139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698034.323141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698034.323142:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698034.323145:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5936 00000100:00080000:3.0:1787698034.323146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698034.323148:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698034.323152:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698034.323158:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134524160:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698034.323160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134524416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698034.323172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698034.323174:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698034.323175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698034.323177:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5936 00000100:00080000:3.0:1787698034.323180:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698034.323183:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698034.323186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698034.323194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134524544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698034.323200:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698034.323202:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698034.323203:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698034.323206:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5936 00000100:00080000:3.0:1787698034.323208:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698034.323211:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698034.323213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698034.323218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134524672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698034.323223:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698034.323226:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698034.323227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698034.323229:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5936 00000100:00080000:3.0:1787698034.323231:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698034.323233:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698034.323236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698034.323241:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698034.323246:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134524800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698034.323257:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698034.323259:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698034.323261:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698034.323263:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5936 00000100:00080000:3.0:1787698034.323265:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698034.323267:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698034.323269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698034.323276:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134524928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698034.323306:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698034/real 1787698034] req@ffff9d85c0fb3b80 x1874531134524160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698034.323325:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134524160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698034.323336:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698034.323340:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698034.323343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134524160:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698034.323350:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134524288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698034.323367:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698034.323375:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698034/real 1787698034] req@ffff9d85c0fb0380 x1874531134524288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698034.323382:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134524288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698034.323388:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698034.323412:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698034.323416:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134524288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698034.323436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134524416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698034.323451:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698034.323460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698034/real 1787698034] req@ffff9d85c0fb2d80 x1874531134524416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698034.323472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134524416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698034.323479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698034.323482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698034.323485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134524416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698034.323490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134524544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698034.323507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698034.323517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698034/real 1787698034] req@ffff9d85c0fb3100 x1874531134524544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698034.323528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134524544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698034.323537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698034.323541:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698034.323544:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134524544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698034.323549:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524672:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698034.323565:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698034.323583:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698034/real 1787698034] req@ffff9d85c0fb2a00 x1874531134524672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698034.323595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134524672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698034.323603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698034.323606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698034.323610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698034.323615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698034.323632:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698034.323642:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698034/real 1787698034] req@ffff9d85c0fb2680 x1874531134524800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698034.323653:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134524800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698034.323661:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698034.323664:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698034.323668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698034.323673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698034.323688:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698034.323698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698034/real 1787698034] req@ffff9d85c0fb0e00 x1874531134524928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698034.323709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134524928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698034.323717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698034.323720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698034.323723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134524928:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698034.899019:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698035.923085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698036.946991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698037.971065:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698038.995013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698039.442943:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698039.442950:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698039.442954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698039.442961:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5941 00000100:00080000:3.0:1787698039.442976:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698039.442982:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698039.443010:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134525056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698039.443031:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698039.443033:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698039.443034:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698039.443037:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5941 00000100:00080000:3.0:1787698039.443040:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698039.443042:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698039.443049:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698039.443057:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134525184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698039.443064:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698039.443066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698039.443067:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698039.443069:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5941 00000100:00080000:3.0:1787698039.443071:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698039.443076:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134525056:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698039.443080:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698039.443088:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134525312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698039.443094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698039.443097:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698039.443098:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698039.443100:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5941 00000100:00080000:3.0:1787698039.443102:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698039.443105:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698039.443107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698039.443114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134525440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698039.443123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698039.443125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698039.443126:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698039.443128:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5941 00000100:00080000:3.0:1787698039.443130:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698039.443134:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698039.443137:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698039.443141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698039.443148:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134525568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698039.443154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698039.443156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698039.443157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698039.443161:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5941 00000100:00080000:3.0:1787698039.443165:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698039.443171:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443172:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698039/real 1787698039] req@ffff9d85c0fb0a80 x1874531134525056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698039.443178:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698039.443194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134525696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698039.443199:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134525056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698039.443213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698039.443217:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698039.443217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698039.443219:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698039.443222:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698039.443226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5941 00000100:00100000:1.0:1787698039.443230:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134525056:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698039.443232:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698039.443238:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698039.443241:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698039.443247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134525184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698039.443251:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134525824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698039.443274:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443285:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698039/real 1787698039] req@ffff9d85c0fb3800 x1874531134525184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698039.443293:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134525184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698039.443298:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698039.443300:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698039.443322:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134525184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698039.443339:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134525312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698039.443353:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698039/real 1787698039] req@ffff9d85c0fb1c00 x1874531134525312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698039.443367:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134525312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698039.443372:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698039.443374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698039.443376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134525312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698039.443379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134525440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698039.443395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443401:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698039/real 1787698039] req@ffff9d85c0fb0000 x1874531134525440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698039.443410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134525440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698039.443418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698039.443422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698039.443425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134525440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698039.443430:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134525568:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698039.443448:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698039/real 1787698039] req@ffff9d85c0fb1f80 x1874531134525568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698039.443469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134525568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698039.443474:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698039.443476:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698039.443477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134525568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698039.443481:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134525696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698039.443493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443499:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698039/real 1787698039] req@ffff9d85c0fb1880 x1874531134525696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698039.443516:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134525696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698039.443520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698039.443522:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698039.443524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134525696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698039.443527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134525824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698039.443537:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698039.443542:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698039/real 1787698039] req@ffff9d85c0fb1500 x1874531134525824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698039.443548:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134525824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698039.443552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698039.443554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698039.443555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134525824:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698040.019020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698041.043047:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698042.067066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698043.091010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698044.114984:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698044.562962:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698044.562969:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698044.562971:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698044.562978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5946 00000100:00080000:0.0:1787698044.562986:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698044.562992:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698044.563020:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134525952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698044.563039:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698044.563041:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698044.563043:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698044.563045:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5946 00000100:00080000:0.0:1787698044.563047:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698044.563049:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698044.563056:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698044.563064:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134526080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698044.563071:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698044.563073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698044.563074:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698044.563077:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5946 00000100:00080000:0.0:1787698044.563080:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698044.563083:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134525952:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698044.563084:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698044.563090:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698044.563096:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134526208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698044.563102:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698044.563105:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698044.563106:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698044.563108:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5946 00000100:00080000:0.0:1787698044.563110:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698044.563112:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698044.563115:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698044.563122:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134526336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698044.563127:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698044.563129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698044.563130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698044.563132:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5946 00000100:00080000:0.0:1787698044.563135:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698044.563137:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698044.563139:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698044.563146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134526464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698044.563156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698044.563158:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698044.563159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698044.563161:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5946 00000100:00080000:0.0:1787698044.563163:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698044.563165:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698044.563168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698044.563175:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134526592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698044.563178:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526080:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698044.563185:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698044.563188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698044.563189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698044.563192:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5946 00000100:00080000:0.0:1787698044.563194:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698044.563197:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698044.563200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698044.563210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134526720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698044.563236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698044.563262:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698044.563279:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134526464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698044.563297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134526592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698044.563315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134526720:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698045.139001:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698046.162980:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698047.186958:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698047.635057:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698047.635090:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698047.635102:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698047.635136:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698047.635146:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698047.635162:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698047.635183:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698047.635190:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698044/real 1787698047] req@ffff9d85b9d0b800 x1874531134526208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698047.635194:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698047.635207:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698047.635230:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134526208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698047.635240:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698047.635244:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698047.635249:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698047.635255:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698044/real 1787698047] req@ffff9d85b9d08700 x1874531134526336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698047.635267:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134526336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698047.635274:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698047.635277:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698047.635280:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698047.635290:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698044/real 1787698047] req@ffff9d85b9d09500 x1874531134526464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698047.635305:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134526464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698047.635313:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698047.635315:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698047.635318:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134526464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698047.635324:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698044/real 1787698047] req@ffff9d85b9d09180 x1874531134526592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698047.635334:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134526592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698047.635341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698047.635344:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698047.635346:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134526592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698047.635352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698044/real 1787698047] req@ffff9d85b9d0a300 x1874531134526720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698047.635363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134526720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698047.635370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698047.635374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698047.635376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134526720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698047.635382:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698044/real 1787698047] req@ffff9d85b9d09f80 x1874531134525952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698047.635392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134525952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698047.635399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698047.635402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698047.635404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134525952:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698047.635410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698044/real 1787698047] req@ffff9d85b9d09c00 x1874531134526080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698047.635419:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134526080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698047.635426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698047.635429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698047.635431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526080:192.168.201.137@tcp:38:kworker.0 00010000:00010000:2.0:1787698048.211002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698049.234954:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698049.682960:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698049.682968:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698049.682976:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698049.682987:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5951 00000100:00080000:3.0:1787698049.682996:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698049.683002:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698049.683036:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134526848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698049.683069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698049.683073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698049.683075:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698049.683079:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5951 00000100:00080000:3.0:1787698049.683083:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698049.683087:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698049.683095:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698049.683108:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134526976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698049.683118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698049.683121:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698049.683122:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698049.683126:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5951 00000100:00080000:3.0:1787698049.683157:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698049.683161:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698049.683166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698049.683176:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134527104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698049.683185:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698049.683188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698049.683189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698049.683192:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5951 00000100:00080000:3.0:1787698049.683195:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698049.683198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698049.683202:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698049.683212:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134527232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698049.683219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698049.683221:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698049.683223:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698049.683226:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5951 00000100:00080000:3.0:1787698049.683229:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698049.683232:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698049.683236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698049.683244:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134527360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698049.683251:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698049.683253:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698049.683255:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698049.683258:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5951 00000100:00080000:3.0:1787698049.683261:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698049.683264:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698049.683267:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698049.683275:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134527488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698049.683282:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698049.683285:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698049.683286:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698049.683289:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5951 00000100:00080000:3.0:1787698049.683291:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698049.683294:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698049.683297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698049.683304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134527616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698049.683397:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134526848:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698049.683452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698049.683502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698049/real 1787698049] req@ffff9d85c0fb2300 x1874531134526848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698049.683520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134526848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698049.683527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698049.683530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698049.683534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134526848:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698049.683541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526976:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698049.683570:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698049.683582:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698049/real 1787698049] req@ffff9d85c0fb3480 x1874531134526976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698049.683593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134526976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698049.683612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698049.683615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698049.683619:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134526976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698049.683625:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134527104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698049.683660:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698049.683672:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698049/real 1787698049] req@ffff9d85c0fb0700 x1874531134527104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698049.683682:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134527104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698049.683689:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698049.683692:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698049.683695:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134527104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698049.683701:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134527232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698049.683718:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698049.683729:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698049/real 1787698049] req@ffff9d85c0fb1500 x1874531134527232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698049.683739:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134527232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698049.683745:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698049.683748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698049.683751:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134527232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698049.683763:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134527360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698049.683781:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698049.683791:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698049/real 1787698049] req@ffff9d85c0fb1880 x1874531134527360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698049.683800:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134527360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698049.683806:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698049.683808:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698049.683811:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134527360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698049.683816:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134527488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698049.683832:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698049.683842:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698049/real 1787698049] req@ffff9d85c0fb1f80 x1874531134527488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698049.683878:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134527488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698049.683887:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698049.683890:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698049.683892:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134527488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698049.683898:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134527616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698049.683915:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698049.683926:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698049/real 1787698049] req@ffff9d85c0fb0000 x1874531134527616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698049.683935:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134527616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698049.683942:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698049.683945:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698049.683947:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134527616:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698050.259003:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698051.283013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698052.307061:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698053.330968:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698054.355006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698054.802922:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698054.802932:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698054.802935:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698054.802945:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5956 00000100:00080000:3.0:1787698054.802954:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698054.802959:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698054.802987:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134527744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698054.803006:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698054.803009:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698054.803010:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698054.803013:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5956 00000100:00080000:3.0:1787698054.803015:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698054.803017:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698054.803024:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698054.803036:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134527872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698054.803063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698054.803066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698054.803068:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698054.803072:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5956 00000100:00080000:3.0:1787698054.803075:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698054.803082:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134527744:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698054.803082:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698054.803087:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698054.803097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134528000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698054.803103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698054.803105:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698054.803106:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698054.803108:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5956 00000100:00080000:3.0:1787698054.803110:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698054.803112:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698054.803114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698054.803121:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134528128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698054.803130:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698054.803133:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698054.803134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698054.803138:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5956 00000100:00080000:3.0:1787698054.803141:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698054.803145:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698054.803149:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698054.803159:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134528256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698054.803168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698054.803173:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698054.803174:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698054.803177:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5956 00000100:00080000:3.0:1787698054.803180:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698054.803182:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698054.803184:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698054.803189:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698054.803197:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134528384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698054.803203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698054.803205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698054.803206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698054.803208:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5956 00000100:00080000:3.0:1787698054.803210:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698054.803211:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698054.803213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698054.803219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134528512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698054.803250:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698054/real 1787698054] req@ffff9d85c0fb1c00 x1874531134527744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698054.803269:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134527744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698054.803293:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698054.803297:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698054.803301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134527744:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698054.803348:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134527872:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698054.803381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698054.803409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698054/real 1787698054] req@ffff9d85c0fb3800 x1874531134527872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698054.803441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134527872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698054.803469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698054.803486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698054.803502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134527872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698054.803510:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698054.803607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698054.803631:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698054/real 1787698054] req@ffff9d85c0fb0a80 x1874531134528000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698054.803650:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134528000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698054.803658:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698054.803662:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698054.803665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698054.803672:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698054.803702:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698054.803712:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698054/real 1787698054] req@ffff9d85c0fb0e00 x1874531134528128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698054.803733:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134528128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698054.803740:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698054.803743:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698054.803747:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698054.803759:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134528256:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698054.803774:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698054.803796:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698054/real 1787698054] req@ffff9d85c0fb2680 x1874531134528256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698054.803824:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134528256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698054.803831:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698054.803834:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698054.803837:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134528256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698054.803890:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134528384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698054.803912:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698054.803935:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698054/real 1787698054] req@ffff9d85c0fb2a00 x1874531134528384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698054.803943:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134528384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698054.803947:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698054.803949:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698054.803951:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134528384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698054.803954:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134528512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698054.803972:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698054.803979:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698054/real 1787698054] req@ffff9d85c0fb3100 x1874531134528512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698054.803985:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134528512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698054.803990:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698054.803991:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698054.803993:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134528512:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698055.379066:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698056.402979:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698057.427036:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698058.451006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698059.475047:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698059.923094:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698059.923109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698059.923114:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698059.923128:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5961 00000100:00080000:2.0:1787698059.923141:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698059.923150:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698059.923200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134528640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698059.923238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698059.923243:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698059.923245:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698059.923250:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5961 00000100:00080000:2.0:1787698059.923254:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698059.923258:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698059.923269:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698059.923284:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134528768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698059.923294:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698059.923298:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698059.923300:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698059.923304:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5961 00000100:00080000:2.0:1787698059.923308:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698059.923312:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698059.923317:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698059.923325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134528896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698059.923331:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698059.923333:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698059.923334:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698059.923337:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5961 00000100:00080000:2.0:1787698059.923338:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698059.923340:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698059.923343:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698059.923349:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134529024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698059.923355:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698059.923357:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698059.923359:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698059.923361:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5961 00000100:00080000:2.0:1787698059.923374:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698059.923377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134528640:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698059.923383:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698059.923389:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698059.923402:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134529152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698059.923413:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698059.923416:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698059.923419:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698059.923423:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5961 00000100:00080000:2.0:1787698059.923427:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698059.923431:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698059.923436:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698059.923449:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134529280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698059.923458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698059.923463:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698059.923467:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698059.923470:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698059.923474:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5961 00000100:00080000:2.0:1787698059.923479:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698059.923483:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698059.923488:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698059.923502:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134529408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698059.923522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698059/real 1787698059] req@ffff9d85babd7480 x1874531134528640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.923544:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134528640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.923556:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.923560:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698059.923564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134528640:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698059.923571:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528768:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698059.923591:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698059.923601:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698059/real 1787698059] req@ffff9d85babd5c00 x1874531134528768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.923612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134528768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.923636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.923639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698059.923642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698059.923648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698059.923665:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698059.923675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698059/real 1787698059] req@ffff9d85babd4000 x1874531134528896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.923685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134528896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.923692:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.923696:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698059.923699:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134528896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698059.923704:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698059.923719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698059.923727:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698059/real 1787698059] req@ffff9d85babd4380 x1874531134529024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.923738:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134529024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.923745:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.923747:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698059.923751:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698059.923777:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134529152:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698059.923793:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698059.923802:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698059/real 1787698059] req@ffff9d85babd6300 x1874531134529152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.923812:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134529152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.923820:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.923823:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698059.923826:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134529152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698059.923831:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134529280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698059.923845:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698059.923890:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698059/real 1787698059] req@ffff9d85babd6a00 x1874531134529280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.923901:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134529280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.923909:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.923912:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698059.923915:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134529280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698059.923920:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134529408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698059.923937:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698059.923946:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698059/real 1787698059] req@ffff9d85babd6680 x1874531134529408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.923973:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134529408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.923981:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.923984:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698059.923987:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134529408:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698060.499065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698061.523097:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698062.547568:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698063.571223:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698064.595000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698065.043496:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698065.043506:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698065.043510:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698065.043521:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5966 00000100:00080000:2.0:1787698065.043532:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698065.043541:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698065.043579:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134529536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698065.043603:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698065.043606:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698065.043608:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698065.043613:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5966 00000100:00080000:2.0:1787698065.043618:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698065.043622:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698065.043648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134529536:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698065.043733:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698065.043733:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698065.043757:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134529664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698065.043770:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698065.043774:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698065.043776:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698065.043783:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5966 00000100:00080000:2.0:1787698065.043789:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698065.043796:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698065/real 1787698065] req@ffff9d85babd7800 x1874531134529536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698065.043801:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698065.043812:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698065.043829:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134529792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698065.043839:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134529536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698065.043904:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698065.043909:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698065.043913:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134529536:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698065.043946:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529664:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698065.043973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698065.043980:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698065.043982:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698065.043986:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698065.043994:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5966 00000100:00080000:2.0:1787698065.044001:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698065.044007:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698065/real 1787698065] req@ffff9d85babd5500 x1874531134529664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698065.044009:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698065.044023:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698065.044041:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134529664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698065.044052:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134529920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698065.044118:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698065.044121:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698065.044130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698065.044133:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698065.044136:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698065.044142:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5966 00000100:00100000:1.0:1787698065.044147:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529664:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698065.044150:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698065.044162:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698065.044169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698065.044174:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698065.044188:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134530048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698065.044199:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698065.044202:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698065.044208:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698065.044210:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698065.044216:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698065/real 1787698065] req@ffff9d85babd6d80 x1874531134529792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698065.044223:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5966 00000100:00080000:2.0:1787698065.044230:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698065.044238:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698065.044243:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134529792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698065.044246:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698065.044265:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698065.044269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134530176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698065.044275:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698065.044285:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529792:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698065.044297:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698065.044303:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529920:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698065.044304:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698065.044310:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698065.044318:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5966 00000100:00080000:2.0:1787698065.044321:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698065.044325:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698065.044330:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698065.044339:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698065.044347:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134530304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698065.044358:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698065/real 1787698065] req@ffff9d85babd7800 x1874531134529920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698065.044373:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134529920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698065.044388:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698065.044391:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698065.044395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134529920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698065.044411:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530048:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698065.044432:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698065.044442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698065/real 1787698065] req@ffff9d85babd6680 x1874531134530048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698065.044453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134530048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698065.044459:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698065.044461:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698065.044463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530048:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698065.044467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698065.044480:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698065.044486:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698065/real 1787698065] req@ffff9d85babd6a00 x1874531134530176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698065.044492:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134530176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698065.044497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698065.044499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698065.044500:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698065.044504:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698065.044513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698065.044519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698065/real 1787698065] req@ffff9d85babd6300 x1874531134530304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698065.044525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134530304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698065.044530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698065.044531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698065.044533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530304:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698065.619027:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698066.643018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698067.667010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698068.691020:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698069.715136:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698070.163030:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698070.163042:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698070.163046:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698070.163056:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5971 00000100:00080000:1.0:1787698070.163069:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698070.163077:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698070.163116:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134530432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698070.163148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698070.163152:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698070.163154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698070.163158:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5971 00000100:00080000:1.0:1787698070.163162:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698070.163166:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698070.163177:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698070.163190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134530560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698070.163198:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134530432:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698070.163204:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698070.163208:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698070.163210:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698070.163214:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5971 00000100:00080000:1.0:1787698070.163217:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698070.163221:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698070.163225:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698070.163236:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134530688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698070.163246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698070.163249:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698070.163250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698070.163254:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5971 00000100:00080000:1.0:1787698070.163257:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698070.163260:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698070.163263:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698070.163268:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698070.163275:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134530816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698070.163292:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698070.163296:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698070.163297:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698070.163301:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5971 00000100:00080000:1.0:1787698070.163306:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698070.163313:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698070.163314:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698070/real 1787698070] req@ffff9d85b9d2dc00 x1874531134530432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698070.163318:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698070.163333:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134530944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698070.163347:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698070.163351:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698070.163353:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134530432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698070.163354:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698070.163364:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5971 00000100:00080000:1.0:1787698070.163370:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698070.163375:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698070.163375:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698070.163381:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698070.163386:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698070.163392:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134531072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698070.163395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134530432:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698070.163404:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698070.163408:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698070.163410:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698070.163414:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5971 00000100:00080000:1.0:1787698070.163418:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698070.163422:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698070.163426:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698070.163437:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134530560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698070.163440:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134531200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698070.163470:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698070.163486:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698070/real 1787698070] req@ffff9d85b9d2e680 x1874531134530560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698070.163494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134530560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698070.163499:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698070.163501:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698070.163503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134530560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698070.163507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134530688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698070.163520:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698070.163529:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698070/real 1787698070] req@ffff9d85b9d2d500 x1874531134530688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698070.163540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134530688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698070.163554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698070.163557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698070.163560:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134530688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698070.163565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134530816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698070.163581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698070.163589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698070/real 1787698070] req@ffff9d85b9d2ed80 x1874531134530816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698070.163600:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134530816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698070.163607:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698070.163610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698070.163613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134530816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698070.163619:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530944:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698070.163636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698070.163646:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698070/real 1787698070] req@ffff9d85b9d2c700 x1874531134530944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698070.163656:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134530944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698070.163662:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698070.163665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698070.163668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134530944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698070.163673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698070.163687:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698070.163695:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698070/real 1787698070] req@ffff9d85b9d2c380 x1874531134531072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698070.163704:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134531072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698070.163711:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698070.163713:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698070.163716:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698070.163732:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698070.163749:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698070.163758:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698070/real 1787698070] req@ffff9d85b9d2dc00 x1874531134531200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698070.163769:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134531200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698070.163776:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698070.163796:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698070.163800:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531200:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698070.739464:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698071.763109:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698072.789845:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698073.810976:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698074.835535:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698075.282966:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698075.282976:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698075.282980:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698075.282989:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5977 00000100:00080000:3.0:1787698075.282999:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698075.283005:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698075.283039:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134531328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698075.283064:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698075.283067:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698075.283069:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698075.283073:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5977 00000100:00080000:3.0:1787698075.283076:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698075.283079:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698075.283088:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698075.283100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134531456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698075.283109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698075.283112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698075.283116:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134531328:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698075.283117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698075.283123:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5977 00000100:00080000:3.0:1787698075.283127:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698075.283130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698075.283136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698075.283146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134531584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698075.283155:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698075.283158:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698075.283160:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698075.283163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5977 00000100:00080000:3.0:1787698075.283168:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698075.283171:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698075.283176:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698075.283189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134531712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698075.283191:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698075.283210:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698075.283214:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698075.283216:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698075.283220:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5977 00000100:00080000:3.0:1787698075.283224:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698075.283230:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698075/real 1787698075] req@ffff9d85c0fb2d80 x1874531134531328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698075.283230:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698075.283238:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698075.283251:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134531840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698075.283257:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134531328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698075.283264:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698075.283272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698075.283275:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698075.283279:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5977 00000100:00080000:0.0:1787698075.283283:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698075.283283:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698075.283290:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698075.283292:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787698075.283296:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698075.283299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134531328:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698075.283308:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134531968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698075.283317:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698075.283320:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698075.283322:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698075.283326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134531456:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698075.283327:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5977 00000100:00080000:3.0:1787698075.283333:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698075.283338:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698075.283342:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698075.283352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134532096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698075.283356:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698075.283373:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698075/real 1787698075] req@ffff9d85c0fb0380 x1874531134531456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698075.283395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134531456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698075.283404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698075.283407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698075.283410:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134531456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698075.283416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134531584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698075.283434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698075.283444:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698075/real 1787698075] req@ffff9d85c0fb3b80 x1874531134531584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698075.283456:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134531584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698075.283470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698075.283473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698075.283476:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134531584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698075.283482:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134531712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698075.283496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698075.283513:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698075/real 1787698075] req@ffff9d85c0fb3100 x1874531134531712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698075.283525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134531712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698075.283533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698075.283536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698075.283539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134531712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698075.283545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531840:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698075.283559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698075.283568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698075/real 1787698075] req@ffff9d85c0fb2a00 x1874531134531840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698075.283578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134531840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698075.283586:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698075.283589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698075.283593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698075.283598:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698075.283614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698075.283622:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698075/real 1787698075] req@ffff9d85c0fb2680 x1874531134531968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698075.283633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134531968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698075.283641:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698075.283643:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698075.283647:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134531968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698075.283662:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698075.283678:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698075.283687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698075/real 1787698075] req@ffff9d85c0fb0e00 x1874531134532096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698075.283723:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134532096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698075.283730:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698075.283733:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698075.283736:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532096:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698075.859012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698076.883018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698077.906996:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698078.931014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698079.955034:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698080.402973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698080.402982:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698080.402986:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698080.402997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5982 00000100:00080000:2.0:1787698080.403008:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698080.403014:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698080.403054:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134532224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698080.403075:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698080.403078:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698080.403079:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698080.403083:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5982 00000100:00080000:2.0:1787698080.403087:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698080.403090:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698080.403101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698080.403115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134532352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698080.403119:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134532224:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698080.403133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698080.403136:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698080.403137:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698080.403142:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5982 00000100:00080000:2.0:1787698080.403145:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698080.403148:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698080.403152:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698080.403162:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134532480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698080.403172:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698080.403174:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698080.403175:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698080.403177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5982 00000100:00080000:2.0:1787698080.403179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698080.403182:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698080.403184:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698080.403191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134532608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698080.403198:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698080.403199:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698080.403202:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698080.403205:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698080.403208:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5982 00000100:00080000:2.0:1787698080.403209:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698080.403212:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698080.403215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698080.403220:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134532736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698080.403226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698080.403229:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698080.403230:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698080.403232:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5982 00000100:00080000:2.0:1787698080.403235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698080.403237:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698080.403241:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698080.403248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134532864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698080.403253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698080.403256:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698080/real 1787698080] req@ffff9d85babd4380 x1874531134532224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698080.403264:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698080.403267:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698080.403273:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5982 00000100:00080000:2.0:1787698080.403276:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698080.403278:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698080.403280:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698080.403283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134532224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698080.403291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134532992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698080.403295:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698080.403302:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698080.403307:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134532224:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698080.403328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134532352:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698080.403346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698080.403355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698080/real 1787698080] req@ffff9d85babd4000 x1874531134532352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698080.403362:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134532352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698080.403367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698080.403369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698080.403371:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134532352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698080.403374:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134532480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698080.403384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698080.403389:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698080/real 1787698080] req@ffff9d85babd5c00 x1874531134532480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698080.403396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134532480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698080.403410:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698080.403413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698080.403415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134532480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698080.403418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134532608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698080.403428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698080.403433:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698080/real 1787698080] req@ffff9d85babd7480 x1874531134532608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698080.403440:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134532608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698080.403445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698080.403446:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698080.403448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134532608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698080.403451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532736:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698080.403460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698080.403465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698080/real 1787698080] req@ffff9d85babd5180 x1874531134532736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698080.403472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134532736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698080.403476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698080.403478:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698080.403480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698080.403483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698080.403492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698080.403497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698080/real 1787698080] req@ffff9d85babd4e00 x1874531134532864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698080.403503:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134532864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698080.403508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698080.403509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698080.403511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698080.403514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698080.403522:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698080.403526:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698080/real 1787698080] req@ffff9d85babd7b80 x1874531134532992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698080.403532:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134532992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698080.403536:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698080.403538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698080.403540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134532992:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698080.979050:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698082.003046:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698083.027055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698084.051043:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698085.074982:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698085.523003:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698085.523013:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698085.523018:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698085.523028:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5987 00000100:00080000:3.0:1787698085.523040:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698085.523047:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698085.523087:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134533120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698085.523114:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698085.523117:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698085.523118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698085.523121:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5987 00000100:00080000:3.0:1787698085.523124:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698085.523126:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698085.523134:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698085.523144:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134533248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698085.523150:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698085.523153:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698085.523154:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698085.523156:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5987 00000100:00080000:3.0:1787698085.523158:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698085.523160:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698085.523163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698085.523170:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134533376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698085.523181:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134533120:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698085.523181:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698085.523187:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698085.523193:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698085.523198:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5987 00000100:00080000:3.0:1787698085.523202:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698085.523204:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698085.523207:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698085.523227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134533504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698085.523233:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698085.523235:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698085.523236:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698085.523239:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5987 00000100:00080000:3.0:1787698085.523241:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698085.523244:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698085.523248:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698085.523259:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698085.523259:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134533632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698085.523272:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698085.523280:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698085.523281:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698085.523288:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5987 00000100:00080000:3.0:1787698085.523291:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698085.523295:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698085.523300:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698085.523304:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698085/real 1787698085] req@ffff9d85c0fb0a80 x1874531134533120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698085.523309:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134533760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698085.523352:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698085.523357:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698085.523363:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698085.523364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134533120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698085.523368:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5987 00000100:00080000:3.0:1787698085.523373:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698085.523375:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698085.523378:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698085.523391:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134533888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698085.523393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.523409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698085.523415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134533120:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698085.523424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134533248:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698085.523446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698085.523454:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698085/real 1787698085] req@ffff9d85c0fb3800 x1874531134533248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.523463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134533248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.523471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.523475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698085.523477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134533248:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698085.523501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134533376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698085.523544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698085.523552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698085/real 1787698085] req@ffff9d85c0fb1c00 x1874531134533376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.523559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134533376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.523568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.523570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698085.523572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134533376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698085.523576:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134533504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698085.523594:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698085.523603:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698085/real 1787698085] req@ffff9d85c0fb0000 x1874531134533504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.523614:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134533504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.523622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.523625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698085.523628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134533504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698085.523634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134533632:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698085.523652:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698085.523660:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698085/real 1787698085] req@ffff9d85c0fb1f80 x1874531134533632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.523667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134533632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.523672:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.523673:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698085.523676:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134533632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698085.523680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134533760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698085.523691:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698085.523699:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698085/real 1787698085] req@ffff9d85c0fb1880 x1874531134533760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.523709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134533760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.523716:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.523719:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698085.523722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134533760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698085.523728:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134533888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698085.523746:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698085.523754:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698085/real 1787698085] req@ffff9d85c0fb1500 x1874531134533888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.523760:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134533888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.523765:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.523767:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698085.523769:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134533888:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698086.099016:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698087.123073:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698088.146972:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698089.170950:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698090.195008:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698090.642968:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698090.642975:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698090.642978:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698090.642986:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5992 00000100:00080000:2.0:1787698090.642994:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698090.642999:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698090.643030:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134534016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698090.643048:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698090.643050:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698090.643052:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698090.643054:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5992 00000100:00080000:2.0:1787698090.643056:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698090.643058:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698090.643066:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698090.643078:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134534144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698090.643084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134534016:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698090.643099:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698090.643103:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698090.643106:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698090.643111:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5992 00000100:00080000:2.0:1787698090.643115:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698090.643119:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698090.643124:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698090.643136:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134534272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698090.643142:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698090.643150:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698090.643154:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698090.643156:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698090.643160:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5992 00000100:00080000:2.0:1787698090.643163:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698090.643166:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698090.643171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698090.643180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134534400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698090.643188:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698090/real 1787698090] req@ffff9d85babd7100 x1874531134534016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698090.643190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698090.643197:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698090.643200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698090.643207:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5992 00000100:00080000:2.0:1787698090.643210:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698090.643214:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698090.643217:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134534016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698090.643220:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698090.643233:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134534528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698090.643235:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698090.643242:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698090.643247:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134534016:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698090.643248:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698090.643255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698090.643257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698090.643261:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5992 00000100:00080000:2.0:1787698090.643265:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698090.643269:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134534144:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698090.643271:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698090.643278:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698090.643288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134534656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698090.643291:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698090.643300:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698090.643303:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698090/real 1787698090] req@ffff9d85babd4a80 x1874531134534144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698090.643304:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698090.643307:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698090.643313:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5992 00000100:00080000:2.0:1787698090.643318:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698090.643322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134534144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698090.643323:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698090.643329:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698090.643333:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698090.643337:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787698090.643340:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134534784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698090.643342:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134534144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698090.643353:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134534272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698090.643382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698090.643394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698090/real 1787698090] req@ffff9d85babd4700 x1874531134534272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698090.643406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134534272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698090.643419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698090.643423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698090.643427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134534272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698090.643464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134534400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698090.643479:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698090.643488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698090/real 1787698090] req@ffff9d85babd5f80 x1874531134534400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698090.643499:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134534400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698090.643506:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698090.643509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698090.643512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134534400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698090.643518:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134534528:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698090.643535:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698090.643542:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698090/real 1787698090] req@ffff9d85babd7b80 x1874531134534528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698090.643549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134534528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698090.643553:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698090.643555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698090.643557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134534528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698090.643568:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134534656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698090.643578:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698090.643583:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698090/real 1787698090] req@ffff9d85babd4e00 x1874531134534656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698090.643590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134534656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698090.643594:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698090.643595:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698090.643597:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134534656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698090.643600:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134534784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698090.643608:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698090.643612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698090/real 1787698090] req@ffff9d85babd5180 x1874531134534784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698090.643618:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134534784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698090.643622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698090.643624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698090.643625:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134534784:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698091.219035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698092.243051:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698093.267013:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698094.291035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698095.314976:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698095.762985:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698095.762998:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698095.763003:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698095.763016:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 5997 00000100:00080000:2.0:1787698095.763029:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698095.763038:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698095.763085:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134534912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698095.763118:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698095.763123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698095.763125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698095.763130:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5997 00000100:00080000:2.0:1787698095.763134:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698095.763139:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698095.763151:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698095.763165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134535040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698095.763170:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134534912:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698095.763182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698095.763186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698095.763188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698095.763193:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5997 00000100:00080000:2.0:1787698095.763197:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698095.763201:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698095.763207:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698095.763219:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134535168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698095.763229:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698095.763233:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698095.763234:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698095.763241:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698095.763241:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 5997 00000100:00080000:2.0:1787698095.763248:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698095.763253:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698095.763259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698095.763272:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134535296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698095.763282:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698095.763286:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698095/real 1787698095] req@ffff9d85babd7480 x1874531134534912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698095.763287:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698095.763292:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698095.763304:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134534912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698095.763309:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5997 00000100:00080000:2.0:1787698095.763315:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698095.763322:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698095.763323:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698095.763332:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787698095.763332:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698095.763351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134534912:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698095.763364:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134535424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698095.763374:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698095.763377:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698095.763379:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698095.763381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535040:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698095.763385:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5997 00000100:00080000:2.0:1787698095.763390:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698095.763394:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698095.763398:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698095.763408:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134535552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698095.763413:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698095.763424:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698095.763427:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698095.763431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698095/real 1787698095] req@ffff9d85babd5c00 x1874531134535040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698095.763431:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698095.763439:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 5997 00000100:00080000:2.0:1787698095.763446:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698095.763452:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698095.763457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134535040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698095.763458:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698095.763471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698095.763474:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134535680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698095.763478:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698095.763486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535040:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698095.763495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698095.763516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698095.763527:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698095/real 1787698095] req@ffff9d85babd4000 x1874531134535168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698095.763539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134535168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698095.763554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698095.763557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698095.763561:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698095.763567:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698095.763583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698095.763592:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698095/real 1787698095] req@ffff9d85babd4380 x1874531134535296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698095.763603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134535296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698095.763611:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698095.763614:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698095.763618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698095.763624:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134535424:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698095.763640:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698095.763649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698095/real 1787698095] req@ffff9d85babd6300 x1874531134535424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698095.763661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134535424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698095.763669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698095.763672:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698095.763676:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134535424:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698095.763693:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134535552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698095.763710:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698095.763720:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698095/real 1787698095] req@ffff9d85babd6a00 x1874531134535552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698095.763731:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134535552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698095.763740:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698095.763743:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698095.763746:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134535552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698095.763752:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134535680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698095.763767:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698095.763777:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698095/real 1787698095] req@ffff9d85babd6680 x1874531134535680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698095.763788:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134535680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698095.763797:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698095.763800:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698095.763803:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134535680:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698096.339079:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698097.363025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698098.387020:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698099.411066:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698100.435030:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698100.882964:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698100.882972:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698100.882975:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698100.882983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6002 00000100:00080000:3.0:1787698100.882991:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698100.882997:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698100.883033:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134535808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698100.883055:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698100.883058:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698100.883059:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698100.883062:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6002 00000100:00080000:3.0:1787698100.883064:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698100.883066:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698100.883073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698100.883080:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134535936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698100.883086:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698100.883089:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698100.883090:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698100.883092:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6002 00000100:00080000:3.0:1787698100.883094:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698100.883096:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698100.883099:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698100.883106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134535808:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698100.883106:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134536064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698100.883118:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698100.883120:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698100.883121:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698100.883123:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6002 00000100:00080000:3.0:1787698100.883125:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698100.883127:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698100.883130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698100.883138:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134536192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698100.883144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698100.883146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698100.883147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698100.883149:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6002 00000100:00080000:3.0:1787698100.883151:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698100.883154:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698100.883156:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698100.883163:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134536320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698100.883167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698100.883174:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698100.883176:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698100.883178:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698100.883181:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6002 00000100:00080000:3.0:1787698100.883184:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698100.883186:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698100.883188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698100.883195:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134536448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698100.883200:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698100.883202:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698100.883204:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698100.883208:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698100/real 1787698100] req@ffff9d85c0fb0700 x1874531134535808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698100.883208:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6002 00000100:00080000:3.0:1787698100.883211:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698100.883216:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698100.883223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698100.883231:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134535808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698100.883232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134536576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698100.883270:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698100.883277:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698100.883281:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134535808:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698100.883290:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535936:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698100.883314:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698100.883326:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698100/real 1787698100] req@ffff9d85c0fb3480 x1874531134535936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698100.883339:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134535936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698100.883347:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698100.883350:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698100.883353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134535936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698100.883375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698100.883393:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698100.883400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698100/real 1787698100] req@ffff9d85c0fb2300 x1874531134536064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698100.883408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134536064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698100.883416:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698100.883418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698100.883420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698100.883423:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698100.883433:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698100.883438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698100/real 1787698100] req@ffff9d85c0fb1500 x1874531134536192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698100.883444:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134536192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698100.883449:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698100.883450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698100.883452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698100.883455:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134536320:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698100.883463:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698100.883468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698100/real 1787698100] req@ffff9d85c0fb1880 x1874531134536320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698100.883474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134536320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698100.883478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698100.883479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698100.883482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134536320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698100.883485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134536448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698100.883493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698100.883498:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698100/real 1787698100] req@ffff9d85c0fb1f80 x1874531134536448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698100.883504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134536448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698100.883508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698100.883510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698100.883511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134536448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698100.883514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134536576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698100.883521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698100.883526:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698100/real 1787698100] req@ffff9d85c0fb0000 x1874531134536576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698100.883533:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134536576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698100.883540:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698100.883553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698100.883557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134536576:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698101.459022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698102.483004:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698103.506969:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698104.530972:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698105.554960:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698106.002904:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698106.002912:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698106.002914:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698106.002921:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6007 00000100:00080000:3.0:1787698106.002929:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698106.002934:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698106.002962:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134536704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698106.002984:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698106.002987:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698106.002989:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698106.002993:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6007 00000100:00080000:3.0:1787698106.002996:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698106.002999:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698106.003007:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698106.003019:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134536832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698106.003029:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698106.003030:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134536704:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698106.003031:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698106.003033:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698106.003036:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6007 00000100:00080000:3.0:1787698106.003039:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698106.003042:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698106.003046:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698106.003053:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134536960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698106.003060:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698106.003063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698106.003064:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698106.003067:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6007 00000100:00080000:3.0:1787698106.003069:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698106.003072:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698106.003075:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698106.003080:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698106.003084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134537088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698106.003090:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698106.003093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698106.003094:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698106.003098:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6007 00000100:00080000:3.0:1787698106.003100:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698106.003103:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698106.003107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698106.003114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134537216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698106.003115:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698106/real 1787698106] req@ffff9d85c0fb1c00 x1874531134536704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698106.003123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698106.003126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698106.003127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698106.003130:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6007 00000100:00100000:0.0:1787698106.003131:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134536704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698106.003133:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698106.003136:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698106.003139:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698106.003139:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698106.003143:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787698106.003146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134537344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698106.003147:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134536704:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698106.003154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698106.003156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698106.003158:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698106.003160:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6007 00000100:00080000:3.0:1787698106.003163:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698106.003179:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536832:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698106.003188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698106.003192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698106.003200:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698106.003201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134537472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698106.003210:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698106/real 1787698106] req@ffff9d85c0fb3800 x1874531134536832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698106.003221:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134536832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698106.003229:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698106.003231:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698106.003235:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698106.003240:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698106.003259:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698106.003269:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698106/real 1787698106] req@ffff9d85c0fb0a80 x1874531134536960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698106.003276:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134536960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698106.003289:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698106.003291:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698106.003293:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134536960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698106.003297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698106.003308:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698106.003314:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698106/real 1787698106] req@ffff9d85c0fb0e00 x1874531134537088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698106.003324:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134537088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698106.003331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698106.003334:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698106.003337:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698106.003342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134537216:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698106.003360:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698106.003369:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698106/real 1787698106] req@ffff9d85c0fb2680 x1874531134537216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698106.003376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134537216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698106.003381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698106.003383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698106.003385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134537216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698106.003388:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134537344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698106.003399:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698106.003405:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698106/real 1787698106] req@ffff9d85c0fb2a00 x1874531134537344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698106.003411:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134537344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698106.003415:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698106.003416:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698106.003418:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134537344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698106.003428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134537472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698106.003437:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698106.003442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698106/real 1787698106] req@ffff9d85c0fb3100 x1874531134537472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698106.003448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134537472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698106.003452:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698106.003454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698106.003455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134537472:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698106.579028:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698107.602996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698108.627010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698109.651015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698110.675004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698111.122945:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698111.122954:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698111.122958:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698111.122966:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6012 00000100:00080000:3.0:1787698111.122977:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698111.122983:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698111.123017:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134537600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698111.123047:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698111.123051:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698111.123052:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698111.123056:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6012 00000100:00080000:3.0:1787698111.123060:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698111.123064:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698111.123072:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698111.123095:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134537728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698111.123106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698111.123108:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134537600:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698111.123111:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698111.123115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698111.123120:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6012 00000100:00080000:3.0:1787698111.123124:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698111.123128:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698111.123133:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698111.123145:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134537856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698111.123154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698111.123157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698111.123158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698111.123162:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6012 00000100:00080000:3.0:1787698111.123165:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698111.123169:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698111.123173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698111.123183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134537984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698111.123191:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698111.123194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698111.123195:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698111.123199:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6012 00000100:00080000:3.0:1787698111.123203:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698111.123209:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537728:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698111.123209:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698111.123217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698111.123231:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134538112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698111.123247:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698111.123249:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537856:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698111.123254:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698111.123257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698111.123262:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6012 00000100:00080000:3.0:1787698111.123267:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698111.123271:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698111.123274:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537984:192.168.201.137@tcp:8:kworker.0 00000008:00100000:3.0:1787698111.123277:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698111.123293:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134538240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698111.123296:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134538112:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698111.123311:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698111.123315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698111.123317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698111.123318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134538240:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698111.123325:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6012 00000100:00080000:3.0:1787698111.123332:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698111.123336:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698111.123342:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698111.123354:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134538368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698111.123391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134538368:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698111.699027:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698112.723004:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698113.746953:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698114.195158:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:1.0:1787698114.195196:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698114.195212:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698114.195249:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698114.195257:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698114.195265:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698114.195272:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698114.195280:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698114.195287:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698114.195349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698111/real 1787698114] req@ffff9d85c0fb0e00 x1874531134538368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698114.195383:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134538368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698114.195394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698114.195398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698114.195413:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134538368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698114.195420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698111/real 1787698114] req@ffff9d85c0fb2680 x1874531134538240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698114.195432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134538240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698114.195439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698114.195442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698114.195444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134538240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698114.195452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698111/real 1787698114] req@ffff9d85c0fb2a00 x1874531134538112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698114.195462:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134538112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698114.195469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698114.195472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698114.195474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134538112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698114.195479:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698111/real 1787698114] req@ffff9d85c0fb0380 x1874531134537728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698114.195489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134537728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698114.195495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698114.195498:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698114.195501:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698114.195506:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698111/real 1787698114] req@ffff9d85c0fb2d80 x1874531134537856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698114.195515:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134537856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698114.195521:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698114.195524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698114.195527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698114.195533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698111/real 1787698114] req@ffff9d85c0fb3100 x1874531134537984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698114.195542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134537984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698114.195548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698114.195550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698114.195553:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134537984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698114.195558:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698111/real 1787698114] req@ffff9d85c0fb3b80 x1874531134537600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698114.195573:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134537600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698114.195579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698114.195582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698114.195584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134537600:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787698114.770958:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698115.795020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698116.242967:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698116.242979:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698116.242982:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698116.243002:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6018 00000100:00080000:2.0:1787698116.243011:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698116.243018:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698116.243044:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134538496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698116.243081:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698116.243084:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698116.243086:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698116.243090:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6018 00000100:00080000:2.0:1787698116.243094:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698116.243098:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698116.243107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698116.243120:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134538624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698116.243130:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698116.243133:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698116.243135:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698116.243138:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6018 00000100:00080000:2.0:1787698116.243142:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698116.243146:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698116.243151:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698116.243159:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134538752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698116.243165:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698116.243167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698116.243168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698116.243170:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6018 00000100:00080000:2.0:1787698116.243173:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698116.243175:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698116.243182:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698116.243193:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134538880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698116.243200:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698116.243204:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698116.243205:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698116.243208:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6018 00000100:00080000:2.0:1787698116.243211:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698116.243215:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698116.243219:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698116.243227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134539008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698116.243236:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698116.243241:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698116.243243:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134538496:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698116.243244:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698116.243252:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6018 00000100:00080000:2.0:1787698116.243255:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698116.243260:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698116.243264:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698116.243276:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134539136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698116.243290:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698116.243293:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698116.243295:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698116.243299:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6018 00000100:00080000:2.0:1787698116.243302:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698116.243305:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698116.243310:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698116.243324:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698116.243325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134539264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698116.243372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698116/real 1787698116] req@ffff9d85babd7800 x1874531134538496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698116.243392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134538496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698116.243404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698116.243409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698116.243421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134538496:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698116.243429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134538624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698116.243453:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698116.243464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698116/real 1787698116] req@ffff9d85babd6d80 x1874531134538624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698116.243476:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134538624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698116.243484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698116.243487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698116.243490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134538624:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698116.243496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134538752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698116.243529:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698116.243541:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698116/real 1787698116] req@ffff9d85babd5500 x1874531134538752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698116.243552:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134538752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698116.243559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698116.243562:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698116.243565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134538752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698116.243571:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134538880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698116.243591:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698116.243602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698116/real 1787698116] req@ffff9d85babd6680 x1874531134538880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698116.243613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134538880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698116.243621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698116.243624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698116.243626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134538880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698116.243654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539008:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698116.243671:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698116.243681:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698116/real 1787698116] req@ffff9d85babd6a00 x1874531134539008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698116.243692:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134539008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698116.243700:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698116.243703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698116.243706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698116.243711:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698116.243726:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698116.243736:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698116/real 1787698116] req@ffff9d85babd6300 x1874531134539136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698116.244078:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134539136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698116.244087:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698116.244090:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698116.244094:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698116.244101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698116.244126:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698116.244139:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698116/real 1787698116] req@ffff9d85babd4380 x1874531134539264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698116.244151:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134539264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698116.244159:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698116.244163:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698116.244166:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539264:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698116.819027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698117.843015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698118.867027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698119.890998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698120.915058:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698121.363009:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698121.363019:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698121.363024:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698121.363035:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6023 00000100:00080000:3.0:1787698121.363049:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698121.363058:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698121.363102:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134539392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698121.363125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698121.363127:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698121.363129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698121.363132:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6023 00000100:00080000:3.0:1787698121.363135:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698121.363137:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698121.363147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698121.363156:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134539520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698121.363162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698121.363165:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698121.363166:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698121.363169:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6023 00000100:00080000:3.0:1787698121.363170:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698121.363172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698121.363176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698121.363183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134539648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698121.363198:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134539392:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698121.363200:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698121.363212:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698121.363215:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698121.363221:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6023 00000100:00080000:3.0:1787698121.363224:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698121.363229:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698121.363235:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698121.363248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134539776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698121.363258:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698121.363262:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698121.363264:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698121.363268:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6023 00000100:00080000:3.0:1787698121.363272:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698121.363279:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698121.363285:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787698121.363289:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698121.363303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134539904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698121.363311:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698121.363314:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698121.363316:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698121.363321:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6023 00000100:00080000:3.0:1787698121.363326:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698121.363329:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698121.363333:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698121.363337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698121/real 1787698121] req@ffff9d85c0fb0a80 x1874531134539392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698121.363343:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134540032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698121.363398:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698121.363402:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698121.363407:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698121.363408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134539392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698121.363421:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6023 00000100:00080000:3.0:1787698121.363433:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698121.363446:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698121.363452:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698121.363454:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698121.363461:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698121.363469:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134539392:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698121.363471:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134540160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698121.363490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134539520:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698121.363516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698121.363528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698121/real 1787698121] req@ffff9d85c0fb3800 x1874531134539520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698121.363540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134539520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698121.363548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698121.363551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698121.363554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134539520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698121.363576:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134539648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698121.363593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698121.363603:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698121/real 1787698121] req@ffff9d85c0fb1c00 x1874531134539648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698121.363613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134539648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698121.363622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698121.363625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698121.363628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134539648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698121.363634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134539776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698121.363650:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698121.363658:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698121/real 1787698121] req@ffff9d85c0fb0000 x1874531134539776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698121.363669:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134539776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698121.363676:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698121.363679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698121.363682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134539776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698121.363687:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539904:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698121.363702:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698121.363712:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698121/real 1787698121] req@ffff9d85c0fb1f80 x1874531134539904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698121.363722:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134539904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698121.363729:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698121.363732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698121.363735:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134539904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698121.363741:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698121.363756:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698121.363765:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698121/real 1787698121] req@ffff9d85c0fb1880 x1874531134540032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698121.363776:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134540032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698121.363783:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698121.363786:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698121.363789:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698121.363794:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698121.363811:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698121.363820:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698121/real 1787698121] req@ffff9d85c0fb1500 x1874531134540160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698121.363830:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134540160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698121.363837:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698121.363840:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698121.363843:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540160:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698121.939044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698122.963079:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698123.987052:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698125.011036:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698126.035073:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698126.482971:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698126.482980:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698126.482984:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698126.482993:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6028 00000100:00080000:0.0:1787698126.483003:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698126.483010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698126.483046:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134540288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698126.483074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698126.483077:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698126.483079:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698126.483083:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6028 00000100:00080000:0.0:1787698126.483087:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698126.483090:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698126.483100:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698126.483114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134540416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698126.483123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698126.483126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698126.483128:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698126.483132:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6028 00000100:00080000:0.0:1787698126.483135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698126.483142:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698126.483149:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698126.483152:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134540288:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698126.483163:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134540544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698126.483177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698126.483181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698126.483183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698126.483186:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6028 00000100:00080000:0.0:1787698126.483201:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698126.483206:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698126.483212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698126.483226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134540672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698126.483235:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698126.483247:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698126.483251:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698126.483253:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698126.483258:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6028 00000100:00080000:0.0:1787698126.483263:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698126.483269:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698126.483277:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698126.483285:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698126/real 1787698126] req@ffff9d85b9d09c00 x1874531134540288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698126.483293:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134540800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698126.483304:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134540288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698126.483307:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698126.483322:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698126.483323:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698126.483332:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698126.483333:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698126.483340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134540288:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698126.483345:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6028 00000100:00100000:1.0:1787698126.483350:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134540416:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698126.483351:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698126.483364:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698126.483372:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698126.483377:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698126.483390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698126/real 1787698126] req@ffff9d85b9d09f80 x1874531134540416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698126.483398:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134540928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698126.483428:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134540416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698126.483472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698126.483473:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698126.483484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698126.483486:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698126.483489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134540416:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698126.483492:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698126.483502:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6028 00000100:00080000:0.0:1787698126.483509:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698126.483513:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698126.483519:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698126.483526:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134540544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698126.483539:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134541056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698126.483556:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698126.483569:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698126/real 1787698126] req@ffff9d85b9d0a300 x1874531134540544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698126.483578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134540544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698126.483583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698126.483586:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698126.483588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134540544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698126.483592:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134540672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698126.483603:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698126.483608:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698126/real 1787698126] req@ffff9d85b9d09180 x1874531134540672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698126.483615:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134540672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698126.483624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698126.483626:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698126.483628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134540672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698126.483631:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540800:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698126.483640:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698126.483645:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698126/real 1787698126] req@ffff9d85b9d09500 x1874531134540800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698126.483651:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134540800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698126.483656:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698126.483657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698126.483659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698126.483662:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698126.483671:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698126.483676:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698126/real 1787698126] req@ffff9d85b9d08700 x1874531134540928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698126.483683:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134540928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698126.483687:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698126.483689:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698126.483691:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134540928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698126.483703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698126.483712:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698126.483717:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698126/real 1787698126] req@ffff9d85b9d0b800 x1874531134541056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698126.483722:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134541056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698126.483727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698126.483729:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698126.483731:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541056:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698127.059012:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698128.083070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698129.107047:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698130.131014:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698131.155089:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698131.602938:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698131.602946:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698131.602949:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698131.602956:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6033 00000100:00080000:2.0:1787698131.602964:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698131.602969:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698131.602996:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134541184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698131.603016:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698131.603018:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698131.603019:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698131.603022:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6033 00000100:00080000:2.0:1787698131.603024:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698131.603026:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698131.603032:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698131.603039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134541312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698131.603045:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698131.603047:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698131.603048:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698131.603051:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6033 00000100:00080000:2.0:1787698131.603052:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698131.603055:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698131.603058:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698131.603068:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134541440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698131.603074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698131.603076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698131.603077:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698131.603080:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6033 00000100:00080000:2.0:1787698131.603082:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698131.603084:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698131.603087:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698131.603094:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134541568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698131.603101:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698131.603105:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698131.603110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698131.603113:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134541184:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698131.603118:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6033 00000100:00080000:2.0:1787698131.603126:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698131.603132:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698131.603137:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698131.603161:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134541696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698131.603167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698131.603171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698131.603172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698131.603177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6033 00000100:00080000:2.0:1787698131.603183:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698131.603187:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698131.603193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698131.603206:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134541824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698131.603210:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698131.603226:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698131.603228:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698131.603230:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698131.603235:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6033 00000100:00080000:2.0:1787698131.603240:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698131.603243:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698131.603247:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698131.603257:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134541952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698131.603287:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698131/real 1787698131] req@ffff9d85babd4000 x1874531134541184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698131.603308:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134541184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698131.603320:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698131.603324:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698131.603328:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134541184:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698131.603337:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134541312:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698131.603363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698131.603377:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698131/real 1787698131] req@ffff9d85babd5c00 x1874531134541312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698131.603389:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134541312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698131.603397:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698131.603400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698131.603404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134541312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698131.603410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134541440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698131.603431:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698131.603443:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698131/real 1787698131] req@ffff9d85babd7480 x1874531134541440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698131.603455:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134541440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698131.603463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698131.603467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698131.603470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134541440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698131.603500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134541568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698131.603521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698131.603533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698131/real 1787698131] req@ffff9d85babd5180 x1874531134541568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698131.603543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134541568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698131.603552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698131.603555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698131.603559:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134541568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698131.603566:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541696:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698131.603585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698131.603595:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698131/real 1787698131] req@ffff9d85babd4e00 x1874531134541696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698131.603605:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134541696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698131.603612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698131.603615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698131.603618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698131.603625:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698131.603642:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698131.603652:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698131/real 1787698131] req@ffff9d85babd7b80 x1874531134541824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698131.603663:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134541824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698131.603671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698131.603674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698131.603677:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698131.603683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698131.603701:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698131.603711:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698131/real 1787698131] req@ffff9d85babd5f80 x1874531134541952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698131.603722:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134541952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698131.603743:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698131.603747:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698131.603751:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134541952:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698132.179021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698133.203076:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698134.226993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698135.251077:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698136.274992:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698136.723000:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698136.723013:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698136.723017:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698136.723029:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6038 00000100:00080000:3.0:1787698136.723043:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698136.723051:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698136.723096:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134542080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698136.723127:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698136.723130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698136.723132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698136.723136:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6038 00000100:00080000:3.0:1787698136.723139:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698136.723142:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698136.723154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698136.723166:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134542208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698136.723177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698136.723180:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134542080:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698136.723181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698136.723185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698136.723192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6038 00000100:00080000:3.0:1787698136.723195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698136.723199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698136.723204:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698136.723215:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134542336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698136.723224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698136.723227:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698136.723229:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698136.723233:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6038 00000100:00080000:3.0:1787698136.723237:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698136.723240:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698136.723245:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698136.723256:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134542464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698136.723264:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698136.723267:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698136.723268:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698136.723274:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6038 00000800:00000100:1.0:1787698136.723277:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698136.723279:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698136.723285:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698136.723290:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698136.723299:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134542592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698136.723307:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698136.723311:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698136.723312:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698136.723316:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6038 00000100:00080000:3.0:1787698136.723320:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698136.723323:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698136.723327:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698136.723337:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134542720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698136.723346:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698136/real 1787698136] req@ffff9d85c0fb2300 x1874531134542080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698136.723350:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698136.723359:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698136.723362:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698136.723368:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6038 00000100:00080000:3.0:1787698136.723373:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698136.723377:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134542080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698136.723379:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698136.723388:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698136.723393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698136.723400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787698136.723404:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134542848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698136.723408:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134542080:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698136.723443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134542208:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698136.723472:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698136.723485:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698136/real 1787698136] req@ffff9d85c0fb3480 x1874531134542208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698136.723505:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134542208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698136.723514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698136.723518:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698136.723521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134542208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698136.723527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134542336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698136.723546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698136.723555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698136/real 1787698136] req@ffff9d85c0fb0700 x1874531134542336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698136.723567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134542336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698136.723574:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698136.723577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698136.723581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134542336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698136.723593:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134542464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698136.723610:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698136.723619:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698136/real 1787698136] req@ffff9d85c0fb1500 x1874531134542464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698136.723631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134542464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698136.723639:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698136.723670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698136.723674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134542464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698136.723680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134542592:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698136.723699:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698136.723710:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698136/real 1787698136] req@ffff9d85c0fb1880 x1874531134542592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698136.723721:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134542592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698136.723729:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698136.723732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698136.723736:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134542592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698136.723742:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134542720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698136.723758:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698136.723767:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698136/real 1787698136] req@ffff9d85c0fb1f80 x1874531134542720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698136.723779:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134542720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698136.723787:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698136.723790:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698136.723794:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134542720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698136.723800:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134542848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698136.723816:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698136.723825:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698136/real 1787698136] req@ffff9d85c0fb0000 x1874531134542848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698136.723837:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134542848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698136.723845:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698136.723848:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698136.723877:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134542848:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698137.299075:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698138.323041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698139.347072:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698140.371056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698141.395088:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698141.842980:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698141.842991:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698141.842994:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698141.843004:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6043 00000100:00080000:2.0:1787698141.843017:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698141.843023:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698141.843062:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134542976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698141.843088:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698141.843092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698141.843094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698141.843098:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6043 00000100:00080000:2.0:1787698141.843101:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698141.843105:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698141.843116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698141.843128:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134543104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698141.843136:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698141.843139:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134542976:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698141.843139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698141.843141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698141.843144:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6043 00000100:00080000:2.0:1787698141.843147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698141.843149:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698141.843155:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698141.843162:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134543232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698141.843169:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698141.843172:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698141.843173:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698141.843176:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6043 00000100:00080000:2.0:1787698141.843179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698141.843182:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698141.843186:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698141.843196:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134543360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698141.843203:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698141.843205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698141.843207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698141.843210:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698141.843210:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6043 00000100:00080000:2.0:1787698141.843213:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698141.843217:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698141.843220:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698141.843228:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134543488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698141.843234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698141.843237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698141.843238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698141.843253:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6043 00000100:00080000:2.0:1787698141.843256:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698141.843260:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698141.843265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698141.843270:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698141/real 1787698141] req@ffff9d85babd4700 x1874531134542976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698141.843273:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134543616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698141.843282:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698141.843284:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698141.843287:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698141.843289:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134542976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698141.843291:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6043 00000100:00080000:2.0:1787698141.843294:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698141.843298:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698141.843300:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698141.843302:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698141.843305:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698141.843310:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134542976:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698141.843310:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134543744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698141.843334:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134543104:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698141.843357:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698141.843367:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698141/real 1787698141] req@ffff9d85babd4a80 x1874531134543104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698141.843375:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134543104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698141.843391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698141.843393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698141.843395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134543104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698141.843399:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134543232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698141.843414:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698141.843424:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698141/real 1787698141] req@ffff9d85babd7100 x1874531134543232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698141.843435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134543232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698141.843443:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698141.843446:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698141.843449:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134543232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698141.843455:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134543360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698141.843473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698141.843482:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698141/real 1787698141] req@ffff9d85babd5f80 x1874531134543360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698141.843491:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134543360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698141.843499:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698141.843502:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698141.843505:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134543360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698141.843510:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134543488:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698141.843524:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698141.843533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698141/real 1787698141] req@ffff9d85babd7b80 x1874531134543488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698141.843543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134543488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698141.843549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698141.843552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698141.843584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134543488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698141.843590:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134543616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698141.843607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698141.843616:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698141/real 1787698141] req@ffff9d85babd4e00 x1874531134543616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698141.843626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134543616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698141.843632:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698141.843635:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698141.843638:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134543616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698141.843643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134543744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698141.843657:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698141.843666:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698141/real 1787698141] req@ffff9d85babd5180 x1874531134543744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698141.843677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134543744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698141.843684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698141.843686:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698141.843689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134543744:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698142.418999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698143.443013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698144.467124:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698145.491018:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698146.514998:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698146.962991:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698146.962999:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698146.963002:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698146.963010:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6048 00000100:00080000:3.0:1787698146.963019:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698146.963024:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698146.963055:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134543872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698146.963076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698146.963079:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698146.963081:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698146.963086:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6048 00000100:00080000:3.0:1787698146.963090:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698146.963093:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698146.963103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698146.963114:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134543872:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698146.963114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134544000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698146.963123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698146.963126:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698146.963127:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698146.963130:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6048 00000100:00080000:3.0:1787698146.963133:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698146.963136:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698146.963140:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698146.963147:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134544128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698146.963154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698146.963157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698146.963159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698146.963161:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6048 00000100:00080000:3.0:1787698146.963164:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698146.963166:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698146.963170:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698146.963180:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698146.963180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134544256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698146.963187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698146.963190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698146.963191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698146.963194:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6048 00000100:00080000:3.0:1787698146.963197:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698146.963200:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698146.963204:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698146.963213:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134544384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698146.963220:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698146.963223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698146.963225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698146.963228:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6048 00000100:00080000:3.0:1787698146.963230:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698146.963233:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698146.963237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698146.963245:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134544512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698146.963251:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698146/real 1787698146] req@ffff9d85c0fb1c00 x1874531134543872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698146.963253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698146.963256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698146.963257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698146.963260:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6048 00000100:00080000:3.0:1787698146.963264:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698146.963267:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698146.963268:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134543872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698146.963271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698146.963278:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698146.963280:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134544640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698146.963282:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698146.963286:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134543872:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698146.963307:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544000:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698146.963327:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698146.963340:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698146/real 1787698146] req@ffff9d85c0fb3800 x1874531134544000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698146.963351:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134544000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698146.963359:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698146.963362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698146.963365:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698146.963370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698146.963387:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698146.963394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698146/real 1787698146] req@ffff9d85c0fb0a80 x1874531134544128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698146.963401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134544128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698146.963405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698146.963407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698146.963409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698146.963412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698146.963421:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698146.963439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698146/real 1787698146] req@ffff9d85c0fb3b80 x1874531134544256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698146.963449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134544256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698146.963457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698146.963460:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698146.963463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698146.963494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134544384:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698146.963513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698146.963524:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698146/real 1787698146] req@ffff9d85c0fb3100 x1874531134544384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698146.963535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134544384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698146.963542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698146.963545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698146.963548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134544384:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698146.963554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134544512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698146.963571:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698146.963580:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698146/real 1787698146] req@ffff9d85c0fb2d80 x1874531134544512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698146.963588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134544512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698146.963596:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698146.963598:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698146.963602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134544512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698146.963606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134544640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698146.963629:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698146.963639:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698146/real 1787698146] req@ffff9d85c0fb0380 x1874531134544640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698146.963649:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134544640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698146.963657:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698146.963659:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698146.963662:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134544640:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698147.539035:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698148.563015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698149.586993:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698150.611004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698151.634955:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698152.082972:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698152.082979:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698152.082981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698152.082988:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6053 00000100:00080000:0.0:1787698152.082997:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698152.083002:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698152.083030:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134544768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698152.083054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698152.083056:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698152.083057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698152.083060:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6053 00000100:00080000:0.0:1787698152.083062:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698152.083064:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698152.083071:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698152.083079:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134544896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698152.083087:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698152.083090:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698152.083092:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698152.083097:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134544768:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698152.083098:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6053 00000100:00080000:0.0:1787698152.083103:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698152.083108:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698152.083114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698152.083125:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134545024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698152.083135:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698152.083138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698152.083139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698152.083142:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6053 00000100:00080000:0.0:1787698152.083152:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698152.083155:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698152.083159:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698152.083169:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698152.083170:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134545152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698152.083178:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698152.083180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698152.083182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698152.083184:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6053 00000100:00080000:0.0:1787698152.083195:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698152.083198:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698152.083201:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698152.083209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134545280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698152.083214:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698152/real 1787698152] req@ffff9d85b9d0b480 x1874531134544768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698152.083218:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698152.083223:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698152.083225:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698152.083231:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6053 00000100:00080000:0.0:1787698152.083236:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698152.083239:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698152.083242:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698152.083248:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134544768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698152.083249:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134545408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698152.083268:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698152.083273:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698152.083274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698152.083276:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698152.083277:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6053 00000100:00080000:0.0:1787698152.083282:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698152.083286:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698152.083287:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698152.083291:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698152.083296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134544768:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698152.083300:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134545536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698152.083325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544896:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698152.083347:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698152.083360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698152/real 1787698152] req@ffff9d85b9d0a680 x1874531134544896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.083410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134544896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.083420:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.083423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698152.083427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134544896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698152.083433:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698152.083460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698152.083472:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698152/real 1787698152] req@ffff9d85b9d0bb80 x1874531134545024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.083483:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134545024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.083491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.083494:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698152.083497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698152.083502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698152.083521:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698152.083531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698152/real 1787698152] req@ffff9d85b9d0ad80 x1874531134545152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.083542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134545152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.083550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.083553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698152.083556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698152.083562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134545280:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698152.083581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698152.083591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698152/real 1787698152] req@ffff9d85b9d08000 x1874531134545280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.083602:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134545280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.083610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.083613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698152.083616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134545280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698152.083622:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134545408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698152.083639:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698152.083649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698152/real 1787698152] req@ffff9d85b9d08a80 x1874531134545408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.083659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134545408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.083667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.083670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698152.083673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134545408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698152.083679:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134545536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698152.083696:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698152.083707:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698152/real 1787698152] req@ffff9d85b9d08e00 x1874531134545536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.083718:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134545536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.083725:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.083728:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698152.083731:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134545536:192.168.201.137@tcp:8:kworker.0 00010000:00010000:0.0:1787698152.659027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698153.682978:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698154.707029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698155.731028:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698156.754959:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698157.202937:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698157.202944:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698157.202947:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698157.202953:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6058 00000100:00080000:0.0:1787698157.202963:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698157.202967:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698157.202999:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134545664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698157.203024:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698157.203028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698157.203029:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698157.203033:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6058 00000100:00080000:0.0:1787698157.203036:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698157.203039:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698157.203045:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698157.203056:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134545792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698157.203063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698157.203066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698157.203069:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698157.203069:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134545664:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698157.203073:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6058 00000100:00080000:0.0:1787698157.203078:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698157.203080:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698157.203083:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698157.203090:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134545920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698157.203096:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698157.203098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698157.203099:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698157.203101:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6058 00000100:00080000:0.0:1787698157.203115:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698157.203117:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698157.203119:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698157.203124:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698157.203127:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134546048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698157.203134:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698157.203138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698157.203140:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698157.203143:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6058 00000100:00080000:0.0:1787698157.203147:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698157.203151:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698157.203156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698157.203157:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698157/real 1787698157] req@ffff9d85b9d08e00 x1874531134545664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698157.203172:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134546176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698157.203173:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134545664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698157.203195:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698157.203201:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698157.203207:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134545664:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698157.203209:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698157.203212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698157.203213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698157.203218:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6058 00000100:00080000:0.0:1787698157.203222:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698157.203224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698157.203228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698157.203236:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134546304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698157.203240:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545792:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698157.203247:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698157.203250:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698157.203251:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698157.203255:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6058 00000100:00080000:0.0:1787698157.203257:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698157.203260:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698157.203264:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698157.203270:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698157.203277:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134546432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698157.203285:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698157/real 1787698157] req@ffff9d85b9d08a80 x1874531134545792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698157.203322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134545792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698157.203331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698157.203334:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698157.203338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698157.203344:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698157.203362:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698157.203370:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698157/real 1787698157] req@ffff9d85b9d08000 x1874531134545920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698157.203376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134545920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698157.203381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698157.203383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698157.203385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134545920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698157.203394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698157.203405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698157.203411:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698157/real 1787698157] req@ffff9d85b9d0ad80 x1874531134546048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698157.203421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134546048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698157.203429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698157.203431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698157.203435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698157.203440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134546176:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698157.203457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698157.203468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698157/real 1787698157] req@ffff9d85b9d0bb80 x1874531134546176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698157.203478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134546176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698157.203486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698157.203488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698157.203491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134546176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698157.203503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134546304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698157.203516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698157.203522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698157/real 1787698157] req@ffff9d85b9d0a680 x1874531134546304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698157.203528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134546304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698157.203533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698157.203534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698157.203537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134546304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698157.203540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134546432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698157.203548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698157.203552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698157/real 1787698157] req@ffff9d85b9d0b480 x1874531134546432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698157.203558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134546432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698157.203562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698157.203564:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698157.203565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134546432:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698157.779024:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698158.803006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698159.827072:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698160.851068:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698161.875029:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698162.323058:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698162.323071:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698162.323075:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698162.323089:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6064 00000100:00080000:3.0:1787698162.323101:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698162.323112:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698162.323363:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134546560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698162.323401:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698162.323406:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698162.323408:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698162.323419:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6064 00000100:00080000:3.0:1787698162.323426:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698162.323431:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698162.323441:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698162.323453:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134546688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698162.323462:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698162.323465:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698162.323467:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698162.323471:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6064 00000100:00080000:3.0:1787698162.323474:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698162.323478:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698162.323483:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698162.323492:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134546816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698162.323501:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698162.323504:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698162.323507:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698162.323510:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6064 00000100:00080000:3.0:1787698162.323514:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698162.323517:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698162.323522:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698162.323532:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134546944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698162.323540:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698162.323544:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698162.323546:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698162.323549:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6064 00000100:00080000:3.0:1787698162.323552:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698162.323556:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698162.323560:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698162.323568:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134547072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698162.323576:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698162.323579:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698162.323580:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698162.323584:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6064 00000100:00080000:3.0:1787698162.323587:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698162.323590:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698162.323594:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698162.323603:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134547200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698162.323610:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698162.323613:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698162.323615:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698162.323618:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6064 00000100:00080000:3.0:1787698162.323621:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698162.323624:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698162.323628:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698162.323637:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134547328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698162.323731:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134546560:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698162.323802:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698162.323902:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698162/real 1787698162] req@ffff9d85c0fb2a00 x1874531134546560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698162.323919:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134546560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698162.323929:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698162.323933:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698162.323937:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134546560:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698162.323945:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546688:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698162.323966:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698162.323979:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698162/real 1787698162] req@ffff9d85c0fb2680 x1874531134546688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698162.323991:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134546688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698162.324000:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698162.324003:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698162.324006:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698162.324012:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698162.324035:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698162.324045:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698162/real 1787698162] req@ffff9d85c0fb0e00 x1874531134546816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698162.324057:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134546816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698162.324065:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698162.324068:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698162.324072:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698162.324077:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698162.324092:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698162.324101:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698162/real 1787698162] req@ffff9d85c0fb0380 x1874531134546944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698162.324112:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134546944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698162.324120:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698162.324123:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698162.324126:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134546944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698162.324132:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547072:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698162.324149:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698162.324159:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698162/real 1787698162] req@ffff9d85c0fb2d80 x1874531134547072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698162.324170:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134547072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698162.324178:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698162.324181:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698162.324184:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698162.324190:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698162.324252:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698162.324264:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698162/real 1787698162] req@ffff9d85c0fb3100 x1874531134547200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698162.324283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134547200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698162.324290:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698162.324293:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698162.324296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698162.324301:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698162.324328:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698162.324337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698162/real 1787698162] req@ffff9d85c0fb3b80 x1874531134547328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698162.324348:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134547328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698162.324356:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698162.324359:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698162.324361:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547328:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698162.899057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698163.923037:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698164.947018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698165.970951:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698166.995030:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698167.442940:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698167.442950:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698167.442964:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698167.442973:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6069 00000100:00080000:3.0:1787698167.442983:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698167.442989:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698167.443018:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134547456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698167.443038:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698167.443040:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698167.443041:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698167.443044:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6069 00000100:00080000:3.0:1787698167.443046:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698167.443048:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698167.443055:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698167.443062:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134547584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698167.443072:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698167.443073:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134547456:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698167.443080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698167.443083:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698167.443087:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6069 00000100:00080000:3.0:1787698167.443091:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698167.443095:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698167.443101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698167.443114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134547712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698167.443139:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698167.443151:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698167.443155:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698167.443157:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698167.443161:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6069 00000100:00080000:3.0:1787698167.443164:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698167.443168:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698167.443172:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698167.443175:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698167/real 1787698167] req@ffff9d85c0fb0a80 x1874531134547456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698167.443184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134547840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698167.443192:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134547456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698167.443197:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698167.443204:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698167.443207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698167.443212:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698167.443213:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6069 00000100:00080000:1.0:1787698167.443220:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698167.443220:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698167.443226:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698167.443229:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134547456:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698167.443231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698167.443243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134547968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698167.443254:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698167.443257:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134547584:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698167.443257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698167.443261:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698167.443267:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6069 00000100:00080000:3.0:1787698167.443269:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698167.443271:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698167.443274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698167.443282:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134548096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698167.443288:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698167.443295:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698167.443299:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698167/real 1787698167] req@ffff9d85c0fb3800 x1874531134547584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698167.443299:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698167.443302:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698167.443309:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6069 00000100:00080000:3.0:1787698167.443314:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698167.443318:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698167.443321:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134547584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698167.443323:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698167.443332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698167.443334:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134548224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698167.443337:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698167.443344:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134547584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698167.443352:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134547712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698167.443365:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698167.443372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698167/real 1787698167] req@ffff9d85c0fb1c00 x1874531134547712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698167.443378:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134547712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698167.443383:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698167.443385:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698167.443387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134547712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698167.443390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134547840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698167.443398:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698167.443403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698167/real 1787698167] req@ffff9d85c0fb0000 x1874531134547840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698167.443409:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134547840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698167.443413:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698167.443415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698167.443417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134547840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698167.443426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547968:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698167.443434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698167.443439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698167/real 1787698167] req@ffff9d85c0fb1f80 x1874531134547968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698167.443445:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134547968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698167.443450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698167.443451:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698167.443453:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134547968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698167.443456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698167.443463:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698167.443468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698167/real 1787698167] req@ffff9d85c0fb1880 x1874531134548096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698167.443474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134548096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698167.443478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698167.443480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698167.443482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698167.443485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698167.443492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698167.443497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698167/real 1787698167] req@ffff9d85c0fb1500 x1874531134548224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698167.443503:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134548224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698167.443507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698167.443509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698167.443510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548224:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698168.019022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698169.042954:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698170.066990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698171.090954:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698172.114956:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698172.562954:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698172.562964:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698172.562968:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698172.562978:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6074 00000100:00080000:3.0:1787698172.562987:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698172.562994:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698172.563041:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134548352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698172.563068:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698172.563072:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698172.563074:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698172.563078:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6074 00000100:00080000:3.0:1787698172.563081:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698172.563084:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698172.563092:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698172.563104:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134548480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698172.563113:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134548352:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698172.563117:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698172.563123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698172.563125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698172.563129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6074 00000100:00080000:3.0:1787698172.563133:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698172.563137:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698172.563143:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698172.563155:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134548608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698172.563169:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698172.563172:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698172.563174:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698172.563177:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6074 00000800:00000100:1.0:1787698172.563182:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698172.563182:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698172.563188:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698172.563193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698172.563204:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134548736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698172.563212:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698172.563215:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698172.563217:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698172.563220:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6074 00000100:00080000:3.0:1787698172.563223:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698172.563226:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698172.563231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698172.563238:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698172/real 1787698172] req@ffff9d85c0fb0700 x1874531134548352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698172.563243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134548864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698172.563260:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698172.563263:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698172.563265:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134548352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698172.563266:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698172.563275:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6074 00000100:00080000:3.0:1787698172.563283:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698172.563285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698172.563288:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698172.563293:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787698172.563294:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698172.563300:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134548352:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698172.563309:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134548992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698172.563319:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698172.563322:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698172.563325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134548480:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698172.563326:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698172.563335:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6074 00000100:00080000:3.0:1787698172.563340:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698172.563344:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698172.563349:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698172.563355:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698172.563363:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134549120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698172.563372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698172/real 1787698172] req@ffff9d85c0fb3480 x1874531134548480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698172.563386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134548480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698172.563395:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698172.563397:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698172.563399:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134548480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698172.563404:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134548608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698172.563429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698172.563440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698172/real 1787698172] req@ffff9d85c0fb2300 x1874531134548608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698172.563452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134548608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698172.563459:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698172.563463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698172.563465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134548608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698172.563469:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134548736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698172.563488:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698172.563495:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698172/real 1787698172] req@ffff9d85c0fb1500 x1874531134548736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698172.563502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134548736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698172.563507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698172.563508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698172.563510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134548736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698172.563514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548864:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698172.563528:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698172.563538:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698172/real 1787698172] req@ffff9d85c0fb1880 x1874531134548864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698172.563546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134548864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698172.563551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698172.563553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698172.563554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698172.563558:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698172.563569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698172.563578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698172/real 1787698172] req@ffff9d85c0fb1f80 x1874531134548992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698172.563588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134548992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698172.563595:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698172.563607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698172.563612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134548992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698172.563628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134549120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698172.563645:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698172.563655:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698172/real 1787698172] req@ffff9d85c0fb0000 x1874531134549120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698172.563664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134549120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698172.563671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698172.563674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698172.563676:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134549120:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698173.138979:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698174.163017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698175.186949:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698176.211000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698177.235034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698177.683015:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698177.683028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698177.683034:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698177.683054:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6079 00000100:00080000:3.0:1787698177.683068:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698177.683077:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698177.683129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134549248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698177.683164:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698177.683169:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698177.683171:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698177.683176:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6079 00000100:00080000:3.0:1787698177.683179:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698177.683184:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698177.683195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698177.683209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134549376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698177.683215:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134549248:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698177.683228:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698177.683232:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698177.683234:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698177.683239:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6079 00000100:00080000:3.0:1787698177.683242:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698177.683246:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698177.683252:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698177.683265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134549504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698177.683274:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698177.683278:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698177.683280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698177.683284:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6079 00000100:00080000:3.0:1787698177.683288:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698177.683292:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698177.683297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698177.683309:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134549632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698177.683333:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698177.683338:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698177.683340:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698177.683347:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6079 00000100:00100000:1.0:1787698177.683352:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134549376:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698177.683353:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698177.683362:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698177.683369:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698177.683392:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134549504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698177.683394:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134549760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698177.683409:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698177.683413:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698177.683415:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698177.683421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134549632:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698177.683422:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6079 00000100:00080000:3.0:1787698177.683429:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698177.683435:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698177.683441:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698177.683461:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134549888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698177.683468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134549760:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698177.683478:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698177.683483:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698177.683486:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698177.683494:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6079 00000100:00080000:3.0:1787698177.683500:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698177.683505:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698177.683510:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134549888:192.168.201.137@tcp:8:kworker.0 00000008:00100000:3.0:1787698177.683513:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698177.683529:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134550016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698177.683544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550016:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698178.259089:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698179.283188:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698180.307034:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698180.755148:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698180.755192:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698180.755208:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698180.755252:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698180.755262:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698180.755269:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698180.755278:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698180.755286:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698180.755295:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698180.755398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698177/real 1787698180] req@ffff9d85c0fb0380 x1874531134550016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698180.755428:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134550016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698180.755440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698180.755444:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698180.755448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698180.755461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698177/real 1787698180] req@ffff9d85c0fb2d80 x1874531134549888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698180.755473:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134549888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698180.755479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698180.755482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698180.755485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134549888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698180.755492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698177/real 1787698180] req@ffff9d85c0fb3100 x1874531134549760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698180.755502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134549760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698180.755509:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698180.755511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698180.755514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134549760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698180.755519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698177/real 1787698180] req@ffff9d85c0fb3800 x1874531134549376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698180.755530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134549376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698180.755537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698180.755540:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698180.755543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134549376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698180.755550:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698177/real 1787698180] req@ffff9d85c0fb0a80 x1874531134549504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698180.755561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134549504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698180.755568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698180.755571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698180.755574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134549504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698180.755580:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698177/real 1787698180] req@ffff9d85c0fb3b80 x1874531134549632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698180.755590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134549632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698180.755597:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698180.755600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698180.755602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134549632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698180.755608:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698177/real 1787698180] req@ffff9d85c0fb1c00 x1874531134549248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698180.755637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134549248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698180.755645:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698180.755648:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698180.755651:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134549248:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787698181.331048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698182.355054:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698182.803001:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698182.803011:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698182.803014:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698182.803023:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6084 00000100:00080000:2.0:1787698182.803032:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698182.803039:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698182.803071:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134550144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698182.803098:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698182.803102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698182.803103:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698182.803108:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6084 00000100:00080000:2.0:1787698182.803111:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698182.803115:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698182.803123:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698182.803136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134550272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698182.803148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698182.803150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134550144:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698182.803154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698182.803157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698182.803163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6084 00000100:00080000:2.0:1787698182.803165:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698182.803169:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698182.803173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698182.803184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134550400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698182.803193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698182.803197:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698182.803198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698182.803202:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6084 00000100:00080000:2.0:1787698182.803205:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698182.803208:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698182.803213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698182.803223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134550528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698182.803232:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698182.803248:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698182.803252:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698182.803255:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698182.803259:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6084 00000100:00080000:2.0:1787698182.803263:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698182.803268:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698182.803272:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698182.803282:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134550656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698182.803290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698182.803293:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698182.803294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698182.803297:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6084 00000100:00080000:2.0:1787698182.803300:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698182.803304:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698182.803309:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698182.803315:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698182/real 1787698182] req@ffff9d85babd7480 x1874531134550144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698182.803322:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134550784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698182.803339:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698182.803343:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698182.803345:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698182.803350:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6084 00000100:00100000:1.0:1787698182.803351:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134550144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698182.803354:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698182.803360:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698182.803368:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698182.803375:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698182.803381:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134550912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698182.803383:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698182.803394:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134550144:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698182.803442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134550272:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698182.803468:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698182.803482:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698182/real 1787698182] req@ffff9d85babd5c00 x1874531134550272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698182.803494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134550272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698182.803502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698182.803506:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698182.803510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134550272:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698182.803516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134550400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698182.803535:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698182.803551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698182/real 1787698182] req@ffff9d85babd4000 x1874531134550400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698182.803564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134550400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698182.803571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698182.803575:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698182.803578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134550400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698182.803584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134550528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698182.803603:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698182.803614:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698182/real 1787698182] req@ffff9d85babd4380 x1874531134550528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698182.803626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134550528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698182.803634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698182.803637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698182.803641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134550528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698182.803646:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550656:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698182.803664:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698182.803675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698182/real 1787698182] req@ffff9d85babd6300 x1874531134550656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698182.803687:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134550656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698182.803693:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698182.803697:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698182.803699:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698182.803703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698182.803735:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698182.803744:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698182/real 1787698182] req@ffff9d85babd6a00 x1874531134550784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698182.803755:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134550784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698182.803763:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698182.803765:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698182.803768:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698182.803773:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698182.803798:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698182.803806:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698182/real 1787698182] req@ffff9d85babd6680 x1874531134550912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698182.803823:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134550912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698182.803830:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698182.803833:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698182.803836:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134550912:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698183.379006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698184.403015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698185.426945:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698186.451059:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698187.475054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698187.923008:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698187.923022:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698187.923028:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698187.923040:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6089 00000100:00080000:2.0:1787698187.923061:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698187.923071:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698187.923127:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134551040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698187.923160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698187.923163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698187.923165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698187.923169:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6089 00000100:00080000:2.0:1787698187.923172:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698187.923175:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698187.923184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698187.923199:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134551168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698187.923210:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698187.923211:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134551040:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698187.923218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698187.923221:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698187.923227:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6089 00000100:00080000:2.0:1787698187.923231:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698187.923235:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698187.923240:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698187.923254:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134551296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698187.923264:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698187.923268:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698187.923270:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698187.923273:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6089 00000100:00080000:2.0:1787698187.923277:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698187.923282:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698187.923285:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698187.923292:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698187.923304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134551424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698187.923311:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698187.923313:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698187.923314:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698187.923317:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6089 00000100:00080000:2.0:1787698187.923320:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698187.923323:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698187.923326:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698187.923332:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134551552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698187.923338:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698187.923340:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698187.923342:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698187.923344:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6089 00000100:00080000:2.0:1787698187.923346:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698187.923350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698187/real 1787698187] req@ffff9d85babd5500 x1874531134551040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698187.923352:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698187.923358:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698187.923369:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134551680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698187.923375:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134551040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698187.923376:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698187.923383:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698187.923386:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698187.923393:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6089 00000100:00080000:1.0:1787698187.923395:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698187.923396:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698187.923401:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698187.923403:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787698187.923406:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698187.923409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134551040:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698187.923416:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134551808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698187.923435:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134551168:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698187.923454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698187.923462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698187/real 1787698187] req@ffff9d85babd6d80 x1874531134551168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698187.923470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134551168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698187.923480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698187.923482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698187.923484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134551168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698187.923487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134551296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698187.923497:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698187.923502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698187/real 1787698187] req@ffff9d85babd7800 x1874531134551296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698187.923509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134551296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698187.923514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698187.923515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698187.923517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134551296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698187.923521:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134551424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698187.923530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698187.923535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698187/real 1787698187] req@ffff9d85babd6680 x1874531134551424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698187.923541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134551424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698187.923546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698187.923547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698187.923550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134551424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698187.923553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134551552:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698187.923562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698187.923567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698187/real 1787698187] req@ffff9d85babd6a00 x1874531134551552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698187.923574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134551552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698187.923578:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698187.923580:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698187.923581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134551552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698187.923584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134551680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698187.923592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698187.923597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698187/real 1787698187] req@ffff9d85babd6300 x1874531134551680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698187.923603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134551680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698187.923608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698187.923609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698187.923611:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134551680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698187.923613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134551808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698187.923621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698187.923626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698187/real 1787698187] req@ffff9d85babd4380 x1874531134551808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698187.923633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134551808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698187.923637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698187.923639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698187.923640:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134551808:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698188.499047:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698189.523052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698190.546953:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698191.570998:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698192.594978:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698193.042955:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698193.042966:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698193.042970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698193.043017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6094 00000100:00080000:2.0:1787698193.043026:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698193.043031:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698193.043062:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134551936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698193.043082:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698193.043084:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698193.043085:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698193.043088:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6094 00000100:00080000:2.0:1787698193.043090:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698193.043092:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698193.043098:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698193.043105:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134552064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698193.043111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698193.043113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698193.043114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698193.043126:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6094 00000100:00080000:2.0:1787698193.043128:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698193.043132:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134551936:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698193.043140:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698193.043145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698193.043154:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134552192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698193.043175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698193.043178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698193.043180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698193.043183:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6094 00000100:00080000:2.0:1787698193.043187:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698193.043191:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698193.043196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698193.043206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134552320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698193.043214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698193.043218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787698193.043222:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698193.043222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698193.043229:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6094 00000100:00080000:2.0:1787698193.043233:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698193.043238:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698193.043243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698193.043254:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134552448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698193.043262:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698193.043266:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698193.043268:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698193.043271:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698193/real 1787698193] req@ffff9d85babd4000 x1874531134551936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698193.043274:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6094 00000100:00080000:2.0:1787698193.043280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698193.043286:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698193.043292:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698193.043299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134551936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698193.043304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134552576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698193.043317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698193.043320:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698193.043325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698193.043326:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698193.043330:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698193.043332:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134551936:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698193.043337:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6094 00000100:00080000:2.0:1787698193.043342:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698193.043343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552064:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698193.043348:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698193.043354:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698193.043366:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134552704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698193.043372:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698193.043389:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698193/real 1787698193] req@ffff9d85babd5c00 x1874531134552064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698193.043402:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134552064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698193.043416:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698193.043428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698193.043432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698193.043476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698193.043495:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698193.043505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698193/real 1787698193] req@ffff9d85babd7480 x1874531134552192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698193.043515:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134552192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698193.043523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698193.043526:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698193.043529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698193.043534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698193.043546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698193.043554:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698193/real 1787698193] req@ffff9d85babd5180 x1874531134552320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698193.043565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134552320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698193.043572:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698193.043575:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698193.043579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698193.043584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134552448:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698193.043602:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698193.043612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698193/real 1787698193] req@ffff9d85babd4e00 x1874531134552448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698193.043622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134552448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698193.043629:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698193.043632:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698193.043635:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134552448:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698193.043640:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134552576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698193.043653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698193.043660:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698193/real 1787698193] req@ffff9d85babd7b80 x1874531134552576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698193.043670:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134552576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698193.043677:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698193.043705:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698193.043709:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134552576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698193.043714:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134552704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698193.043729:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698193.043738:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698193/real 1787698193] req@ffff9d85babd5f80 x1874531134552704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698193.043754:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134552704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698193.043762:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698193.043764:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698193.043767:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134552704:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698193.619024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698194.643038:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698195.666996:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698196.691022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698197.715026:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698198.162952:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698198.162967:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698198.162971:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698198.162984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6099 00000100:00080000:3.0:1787698198.162997:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698198.163005:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698198.163050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134552832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698198.163084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698198.163089:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698198.163091:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698198.163095:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6099 00000100:00080000:3.0:1787698198.163099:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698198.163103:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698198.163115:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698198.163138:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134552960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698198.163146:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134552832:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698198.163152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698198.163162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698198.163164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698198.163170:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6099 00000100:00080000:3.0:1787698198.163173:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698198.163177:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698198.163184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698198.163195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134553088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698198.163202:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698198.163204:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698198.163206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698198.163210:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6099 00000100:00080000:3.0:1787698198.163214:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698198.163216:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698198.163219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698198.163228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134553216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698198.163241:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698198.163251:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698198.163254:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698198.163256:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698198.163266:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6099 00000100:00080000:3.0:1787698198.163275:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698198.163281:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698198.163288:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698198/real 1787698198] req@ffff9d85c0fb0e00 x1874531134552832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698198.163288:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698198.163307:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134553344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698198.163316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134552832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698198.163321:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698198.163332:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698198.163336:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698198.163347:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6099 00000100:00080000:3.0:1787698198.163351:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698198.163353:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698198.163358:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698198.163367:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787698198.163368:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698198.163380:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134552832:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698198.163384:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134553472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698198.163399:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698198.163402:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698198.163404:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698198.163406:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6099 00000100:00080000:3.0:1787698198.163409:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698198.163412:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698198.163415:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698198.163422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698198.163423:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134553600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698198.163463:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698198.163491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698198/real 1787698198] req@ffff9d85c0fb2680 x1874531134552960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698198.163503:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134552960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698198.163516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698198.163527:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698198.163531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134552960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698198.163541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698198.163560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698198.163571:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698198/real 1787698198] req@ffff9d85c0fb2a00 x1874531134553088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698198.163581:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134553088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698198.163588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698198.163602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698198.163606:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698198.163612:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698198.163632:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698198.163642:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698198/real 1787698198] req@ffff9d85c0fb1c00 x1874531134553216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698198.163649:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134553216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698198.163655:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698198.163657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698198.163659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698198.163663:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134553344:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698198.163675:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698198.163681:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698198/real 1787698198] req@ffff9d85c0fb3b80 x1874531134553344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698198.163687:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134553344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698198.163691:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698198.163693:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698198.163695:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134553344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698198.163698:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134553472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698198.163707:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698198.163712:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698198/real 1787698198] req@ffff9d85c0fb0a80 x1874531134553472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698198.163718:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134553472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698198.163722:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698198.163724:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698198.163726:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134553472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698198.163737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134553600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698198.163746:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698198.163751:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698198/real 1787698198] req@ffff9d85c0fb3800 x1874531134553600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698198.163758:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134553600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698198.163762:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698198.163764:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698198.163765:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134553600:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698198.739097:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698199.763021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698200.787001:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698201.811031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698202.835030:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698203.282957:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698203.282964:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698203.282967:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698203.282974:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6105 00000100:00080000:1.0:1787698203.282983:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698203.282988:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698203.283020:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134553728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698203.283050:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698203.283054:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698203.283056:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698203.283059:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6105 00000100:00080000:1.0:1787698203.283062:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698203.283064:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698203.283072:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698203.283081:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134553856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698203.283090:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134553728:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698203.283091:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698203.283098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698203.283100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698203.283103:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6105 00000100:00080000:1.0:1787698203.283104:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698203.283106:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698203.283110:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698203.283117:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134553984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698203.283123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698203.283125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698203.283126:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698203.283128:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6105 00000100:00080000:1.0:1787698203.283143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698203.283146:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698203.283148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698203.283155:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698203.283156:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134554112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698203.283163:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698203.283165:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698203.283167:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698203.283176:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6105 00000100:00080000:1.0:1787698203.283179:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698203.283181:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698203.283184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698203.283192:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134554240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698203.283200:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698203/real 1787698203] req@ffff9d85b9d2f800 x1874531134553728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698203.283200:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698203.283203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698203.283206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698203.283212:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6105 00000100:00080000:1.0:1787698203.283217:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698203.283221:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698203.283224:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698203.283234:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134554368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698203.283236:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134553728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698203.283245:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698203.283251:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698203.283254:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698203.283260:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6105 00000100:00080000:0.0:1787698203.283262:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698203.283262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698203.283265:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698203.283269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698203.283272:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698203.283277:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134554496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698203.283281:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134553728:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698203.283311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553856:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698203.283335:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698203.283346:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698203/real 1787698203] req@ffff9d85b9d2fb80 x1874531134553856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698203.283358:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134553856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698203.283366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698203.283369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698203.283373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698203.283379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698203.283397:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698203.283408:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698203/real 1787698203] req@ffff9d85b9d2e300 x1874531134553984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698203.283425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134553984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698203.283433:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698203.283436:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698203.283440:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134553984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698203.283454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134554112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698203.283474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698203.283484:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698203/real 1787698203] req@ffff9d85b9d2ce00 x1874531134554112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698203.283495:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134554112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698203.283502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698203.283530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698203.283534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134554112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698203.283540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134554240:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698203.283560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698203.283570:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698203/real 1787698203] req@ffff9d85b9d2c000 x1874531134554240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698203.283581:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134554240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698203.283594:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698203.283597:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698203.283600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134554240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698203.283606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134554368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698203.283624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698203.283634:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698203/real 1787698203] req@ffff9d85b9d2ca80 x1874531134554368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698203.283645:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134554368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698203.283653:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698203.283656:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698203.283659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134554368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698203.283664:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134554496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698203.283682:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698203.283692:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698203/real 1787698203] req@ffff9d85b9d2f480 x1874531134554496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698203.283703:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134554496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698203.283711:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698203.283714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698203.283717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134554496:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698203.859075:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698204.883053:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698205.907005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698206.930954:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698207.955015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698208.402983:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698208.402994:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698208.402998:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698208.403007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6110 00000100:00080000:3.0:1787698208.403016:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698208.403023:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698208.403067:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134554624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698208.403099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698208.403103:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698208.403105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698208.403109:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6110 00000100:00080000:3.0:1787698208.403112:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698208.403116:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698208.403124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698208.403135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134554752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698208.403144:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698208.403147:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698208.403148:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698208.403151:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6110 00000100:00080000:3.0:1787698208.403154:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698208.403157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698208.403161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698208.403168:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134554880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698208.403177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698208.403180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698208.403181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698208.403184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6110 00000100:00080000:3.0:1787698208.403186:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698208.403189:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698208.403192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698208.403200:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134555008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698208.403207:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698208.403210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698208.403211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698208.403213:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6110 00000100:00080000:3.0:1787698208.403216:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698208.403219:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698208.403223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698208.403229:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134555136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698208.403236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698208.403239:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698208.403240:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698208.403243:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6110 00000100:00080000:3.0:1787698208.403246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698208.403248:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698208.403251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698208.403258:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134555264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698208.403265:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698208.403267:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698208.403269:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698208.403271:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6110 00000100:00080000:3.0:1787698208.403273:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698208.403275:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698208.403278:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698208.403285:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134555392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698208.403375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134554624:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787698208.403443:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698208.403472:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698208/real 1787698208] req@ffff9d85c0fb3100 x1874531134554624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698208.403483:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134554624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698208.403489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698208.403492:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698208.403494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134554624:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698208.403498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134554752:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698208.403511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698208.403518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698208/real 1787698208] req@ffff9d85c0fb2d80 x1874531134554752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698208.403524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134554752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698208.403529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698208.403531:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698208.403532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134554752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698208.403535:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134554880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698208.403544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698208.403550:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698208/real 1787698208] req@ffff9d85c0fb0380 x1874531134554880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698208.403556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134554880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698208.403569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698208.403571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698208.403572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134554880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698208.403576:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698208.403586:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698208.403591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698208/real 1787698208] req@ffff9d85c0fb0000 x1874531134555008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698208.403598:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134555008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698208.403602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698208.403604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698208.403607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698208.403612:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134555136:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698208.403628:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698208.403638:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698208/real 1787698208] req@ffff9d85c0fb1f80 x1874531134555136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698208.403647:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134555136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698208.403663:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698208.403666:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698208.403669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134555136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698208.403674:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134555264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698208.403690:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698208.403699:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698208/real 1787698208] req@ffff9d85c0fb1880 x1874531134555264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698208.403709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134555264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698208.403715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698208.403718:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698208.403721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134555264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698208.403725:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134555392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698208.403747:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698208.403756:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698208/real 1787698208] req@ffff9d85c0fb1500 x1874531134555392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698208.403765:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134555392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698208.403772:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698208.403774:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698208.403777:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134555392:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698208.981558:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698210.002993:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698211.027024:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698212.050958:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698213.075020:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698213.522961:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698213.522970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698213.522974:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698213.522984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6115 00000100:00080000:3.0:1787698213.522993:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698213.522999:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698213.523075:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134555520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698213.523104:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698213.523108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698213.523110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698213.523114:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6115 00000100:00080000:3.0:1787698213.523117:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698213.523121:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698213.523129:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698213.523141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134555648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698213.523151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698213.523154:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134555520:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698213.523155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698213.523159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698213.523165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6115 00000100:00080000:3.0:1787698213.523168:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698213.523172:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698213.523178:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698213.523189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134555776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698213.523197:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698213.523200:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698213.523202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698213.523205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6115 00000100:00080000:3.0:1787698213.523208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698213.523213:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698213.523218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698213.523221:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698213.523233:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134555904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698213.523242:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698213.523245:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698213.523246:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698213.523249:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6115 00000100:00080000:3.0:1787698213.523252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698213.523255:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698213.523259:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698213.523268:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134556032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698213.523280:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698213.523283:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698213.523285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698213.523288:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6115 00000100:00080000:3.0:1787698213.523291:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698213.523294:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698213.523298:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698213.523306:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134556160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698213.523313:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698213.523316:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698213.523318:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698213.523321:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6115 00000100:00080000:3.0:1787698213.523323:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698213.523327:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698213.523342:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698213.523352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134556288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698213.524344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698213/real 1787698213] req@ffff9d85c0fb2300 x1874531134555520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698213.524359:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134555520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698213.524366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698213.524369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698213.524372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134555520:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698213.524388:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555648:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698213.524406:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698213.524413:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698213/real 1787698213] req@ffff9d85c0fb3480 x1874531134555648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698213.524421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134555648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698213.524425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698213.524427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698213.524429:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698213.524432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698213.524441:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698213.524447:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698213/real 1787698213] req@ffff9d85c0fb0700 x1874531134555776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698213.524453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134555776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698213.524457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698213.524459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698213.524460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698213.524463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698213.524472:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698213.524476:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698213/real 1787698213] req@ffff9d85c0fb1500 x1874531134555904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698213.524482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134555904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698213.524487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698213.524488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698213.524490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134555904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698213.524493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556032:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698213.524501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698213.524505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698213/real 1787698213] req@ffff9d85c0fb1880 x1874531134556032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698213.524511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134556032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698213.524516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698213.524517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698213.524519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698213.524522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698213.524530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698213.524534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698213/real 1787698213] req@ffff9d85c0fb1f80 x1874531134556160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698213.524540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134556160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698213.524544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698213.524545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698213.524547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698213.524550:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698213.524558:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698213.524562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698213/real 1787698213] req@ffff9d85c0fb0000 x1874531134556288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698213.524567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134556288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698213.524571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698213.524573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698213.524575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556288:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698214.099038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698215.122967:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698216.146985:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698217.170973:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698218.195025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698218.642970:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698218.642980:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698218.642984:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698218.642993:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6120 00000100:00080000:3.0:1787698218.643003:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698218.643010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698218.643044:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134556416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698218.643070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698218.643072:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698218.643073:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698218.643076:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6120 00000100:00080000:3.0:1787698218.643079:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698218.643081:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698218.643089:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698218.643099:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134556544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698218.643106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698218.643110:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698218.643113:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134556416:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698218.643113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698218.643122:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6120 00000100:00080000:3.0:1787698218.643126:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698218.643130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698218.643136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698218.643150:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134556672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698218.643165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698218.643168:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698218.643171:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698218.643175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698218.643180:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6120 00000100:00080000:3.0:1787698218.643184:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698218.643188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698218.643193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698218.643203:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698218/real 1787698218] req@ffff9d85c0fb0380 x1874531134556416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698218.643207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134556800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698218.643228:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134556416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698218.643233:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698218.643237:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698218.643238:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698218.643257:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698218.643258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698218.643263:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134556416:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698218.643265:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6120 00000100:00080000:3.0:1787698218.643272:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698218.643277:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698218.643283:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698218.643286:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134556544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698218.643300:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134556928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698218.643309:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698218.643320:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698218.643324:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698218.643326:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698218.643328:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698218/real 1787698218] req@ffff9d85c0fb2d80 x1874531134556544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698218.643331:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6120 00000100:00100000:0.0:1787698218.643339:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134556544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698218.643344:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698218.643346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698218.643348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134556544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698218.643352:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134556672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698218.643363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698218.643369:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698218/real 1787698218] req@ffff9d85c0fb3100 x1874531134556672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698218.643374:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698218.643380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134556672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698218.643383:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698218.643391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698218.643392:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698218.643394:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698218.643399:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134556672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698218.643403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134556800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698218.643411:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134557056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698218.643417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698218.643425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698218/real 1787698218] req@ffff9d85c0fb3800 x1874531134556800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698218.643428:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698218.643436:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698218.643438:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698218.643440:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134556800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698218.643445:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6120 00000100:00080000:3.0:1787698218.643452:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698218.643453:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698218.643458:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698218.643461:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787698218.643465:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698218.643468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134556800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698218.643478:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134557184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698218.643489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556928:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698218.643511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698218.643522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698218/real 1787698218] req@ffff9d85c0fb0a80 x1874531134556928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698218.643533:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134556928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698218.643539:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698218.643541:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698218.643543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134556928:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698218.643546:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698218.643559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698218.643565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698218/real 1787698218] req@ffff9d85c0fb3b80 x1874531134557056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698218.643571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134557056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698218.643575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698218.643577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698218.643578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698218.643586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698218.643594:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698218.643599:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698218/real 1787698218] req@ffff9d85c0fb1c00 x1874531134557184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698218.643605:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134557184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698218.643609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698218.643610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698218.643612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557184:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698219.219049:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698220.243035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698221.267015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698222.290985:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698223.315010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698223.762973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698223.762983:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698223.762988:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698223.762998:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6125 00000100:00080000:0.0:1787698223.763008:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698223.763016:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698223.763052:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134557312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698223.763080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698223.763084:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698223.763085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698223.763089:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6125 00000100:00080000:0.0:1787698223.763093:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698223.763098:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698223.763107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698223.763120:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134557440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698223.763130:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134557312:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698223.763133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698223.763140:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698223.763142:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698223.763146:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6125 00000100:00080000:0.0:1787698223.763149:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698223.763153:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698223.763170:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698223.763183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134557568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698223.763192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698223.763196:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698223.763197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698223.763201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6125 00000100:00080000:0.0:1787698223.763212:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698223.763219:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698223.763224:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698223.763230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698223.763243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134557696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698223.763252:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698223.763255:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698223.763257:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698223.763263:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698223/real 1787698223] req@ffff9d85b9d0b800 x1874531134557312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698223.763265:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6125 00000100:00080000:0.0:1787698223.763271:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698223.763279:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698223.763284:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698223.763291:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134557312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698223.763297:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134557824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698223.763309:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698223.763312:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698223.763316:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698223.763318:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698223.763323:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698223.763324:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134557312:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698223.763329:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6125 00000100:00080000:0.0:1787698223.763334:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698223.763338:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698223.763343:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698223.763353:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134557440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698223.763355:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134557952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698223.763368:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698223.763371:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698223.763372:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698223.763375:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6125 00000100:00080000:0.0:1787698223.763380:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698223.763380:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698223.763387:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698223.763399:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698223/real 1787698223] req@ffff9d85b9d08700 x1874531134557440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698223.763401:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698223.763417:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134558080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698223.763421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134557440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698223.763440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698223.763443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698223.763446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134557440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698223.763450:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134557568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698223.763466:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698223.763473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698223/real 1787698223] req@ffff9d85b9d09500 x1874531134557568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698223.763480:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134557568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698223.763485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698223.763487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698223.763488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134557568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698223.763491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134557696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698223.763500:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698223.763505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698223/real 1787698223] req@ffff9d85b9d09180 x1874531134557696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698223.763511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134557696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698223.763516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698223.763517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698223.763519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134557696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698223.763522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557824:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698223.763534:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698223.763543:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698223/real 1787698223] req@ffff9d85b9d0a300 x1874531134557824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698223.763553:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134557824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698223.763561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698223.763563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698223.763566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557824:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698223.763582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698223.763598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698223.763607:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698223/real 1787698223] req@ffff9d85b9d09f80 x1874531134557952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698223.763618:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134557952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698223.763625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698223.763628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698223.763631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134557952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698223.763636:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698223.763654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698223.763664:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698223/real 1787698223] req@ffff9d85b9d09c00 x1874531134558080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698223.763675:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134558080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698223.763682:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698223.763685:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698223.763688:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558080:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698224.339029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698225.363019:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698226.387024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698227.411021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698228.435000:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698228.882951:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698228.882957:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698228.882960:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698228.882967:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6130 00000100:00080000:0.0:1787698228.882976:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698228.882981:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698228.883011:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134558208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698228.883032:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698228.883035:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698228.883036:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698228.883039:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6130 00000100:00080000:0.0:1787698228.883040:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698228.883043:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698228.883051:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698228.883062:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134558336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698228.883095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698228.883101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134558208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698228.883104:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698228.883108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698228.883114:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6130 00000100:00080000:0.0:1787698228.883118:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698228.883122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698228.883128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698228.883140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134558464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698228.883149:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698228.883153:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698228.883154:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698228.883158:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6130 00000100:00080000:0.0:1787698228.883169:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698228.883174:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698228.883180:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698228.883181:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698228.883196:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134558592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698228.883204:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698228.883207:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698228.883208:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698228.883211:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6130 00000100:00080000:0.0:1787698228.883214:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698228.883217:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698228.883220:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698228.883223:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698228/real 1787698228] req@ffff9d85b9d09c00 x1874531134558208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698228.883230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134558720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698228.883245:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698228.883248:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698228.883249:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698228.883252:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6130 00000100:00080000:0.0:1787698228.883255:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698228.883256:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134558208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698228.883259:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698228.883265:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698228.883273:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698228.883276:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134558848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698228.883280:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698228.883286:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134558208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698228.883297:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698228.883300:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698228.883301:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698228.883304:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6130 00000100:00080000:0.0:1787698228.883306:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698228.883311:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698228.883311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134558336:192.168.201.137@tcp:38:kworker.0 00000008:00100000:0.0:1787698228.883317:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698228.883326:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134558976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698228.883338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698228.883348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698228/real 1787698228] req@ffff9d85b9d09f80 x1874531134558336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698228.883357:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134558336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698228.883363:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698228.883365:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698228.883367:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134558336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698228.883370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134558464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698228.883381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698228.883387:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698228/real 1787698228] req@ffff9d85b9d0a300 x1874531134558464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698228.883393:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134558464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698228.883397:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698228.883399:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698228.883401:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134558464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698228.883406:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134558592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698228.883421:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698228.883430:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698228/real 1787698228] req@ffff9d85b9d09180 x1874531134558592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698228.883442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134558592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698228.883449:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698228.883452:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698228.883454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134558592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698228.883458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558720:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698228.883470:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698228.883477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698228/real 1787698228] req@ffff9d85b9d09500 x1874531134558720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698228.883483:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134558720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698228.883488:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698228.883490:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698228.883492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698228.883494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698228.883503:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698228.883507:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698228/real 1787698228] req@ffff9d85b9d08700 x1874531134558848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698228.883513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134558848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698228.883518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698228.883519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698228.883521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698228.883532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698228.883540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698228.883546:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698228/real 1787698228] req@ffff9d85b9d0b800 x1874531134558976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698228.883552:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134558976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698228.883556:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698228.883557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698228.883559:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134558976:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698229.459038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698230.483024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698231.507008:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698232.531002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698233.554952:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698234.002971:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698234.002996:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698234.003001:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698234.003011:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6135 00000100:00080000:3.0:1787698234.003022:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698234.003029:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698234.003063:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134559104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698234.003086:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698234.003090:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698234.003091:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698234.003096:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6135 00000100:00080000:3.0:1787698234.003099:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698234.003103:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698234.003113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698234.003127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134559232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698234.003131:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134559104:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698234.003138:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698234.003141:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698234.003142:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698234.003186:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6135 00000100:00080000:3.0:1787698234.003192:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698234.003194:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698234.003199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698234.003205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698234.003216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134559360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698234.003223:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698234.003226:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698234.003227:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698234.003230:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6135 00000100:00080000:3.0:1787698234.003233:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698234.003235:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698234.003239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698234.003241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698234/real 1787698234] req@ffff9d85c0fb2a00 x1874531134559104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698234.003250:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134559488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698234.003260:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698234.003263:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698234.003265:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698234.003266:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134559104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698234.003269:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6135 00000100:00080000:3.0:1787698234.003274:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698234.003279:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698234.003284:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698234.003285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698234.003291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134559616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698234.003292:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698234.003300:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134559104:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698234.003304:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698234.003307:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698234.003309:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698234.003312:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6135 00000100:00080000:3.0:1787698234.003314:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698234.003316:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698234.003319:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698234.003325:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134559744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698234.003328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134559232:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698234.003339:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698234.003342:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698234.003343:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698234.003346:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6135 00000100:00080000:3.0:1787698234.003348:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698234.003350:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698234.003353:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698234.003359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698234.003365:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134559872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698234.003374:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698234/real 1787698234] req@ffff9d85c0fb2680 x1874531134559232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698234.003386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134559232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698234.003394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698234.003397:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698234.003400:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134559232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698234.003406:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134559360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698234.003421:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698234.003429:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698234/real 1787698234] req@ffff9d85c0fb0e00 x1874531134559360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698234.003439:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134559360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698234.003446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698234.003449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698234.003452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134559360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698234.003457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134559488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698234.003472:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698234.003480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698234/real 1787698234] req@ffff9d85c0fb1c00 x1874531134559488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698234.003490:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134559488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698234.003497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698234.003500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698234.003503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134559488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698234.003507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134559616:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698234.003523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698234.003531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698234/real 1787698234] req@ffff9d85c0fb3b80 x1874531134559616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698234.003539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134559616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698234.003546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698234.003549:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698234.003551:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134559616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698234.003565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134559744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698234.003580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698234.003588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698234/real 1787698234] req@ffff9d85c0fb0a80 x1874531134559744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698234.003597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134559744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698234.003604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698234.003606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698234.003609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134559744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698234.003626:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134559872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698234.003640:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698234.003649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698234/real 1787698234] req@ffff9d85c0fb3800 x1874531134559872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698234.003658:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134559872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698234.003664:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698234.003666:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698234.003669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134559872:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698234.579112:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698235.603049:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698236.627016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698237.651029:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698238.675055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698239.122967:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698239.122975:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698239.122978:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698239.122985:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6140 00000100:00080000:2.0:1787698239.123000:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698239.123005:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698239.123039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134560000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698239.123059:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698239.123061:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698239.123063:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698239.123066:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6140 00000100:00080000:2.0:1787698239.123068:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698239.123070:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698239.123076:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698239.123085:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134560128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698239.123091:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698239.123094:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698239.123095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698239.123097:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6140 00000100:00080000:2.0:1787698239.123098:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698239.123102:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134560000:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698239.123103:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698239.123108:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698239.123115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134560256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698239.123124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698239.123127:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698239.123129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698239.123133:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6140 00000100:00080000:2.0:1787698239.123143:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698239.123147:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698239.123152:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698239.123166:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698239.123168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134560384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698239.123179:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698239.123182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698239.123184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698239.123187:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6140 00000100:00080000:2.0:1787698239.123190:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698239.123193:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698239.123197:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698239.123206:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134560512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698239.123212:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698239/real 1787698239] req@ffff9d85babd7100 x1874531134560000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698239.123215:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698239.123220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698239.123223:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698239.123231:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6140 00000100:00080000:2.0:1787698239.123235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698239.123238:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134560000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698239.123241:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698239.123250:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698239.123255:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698239.123261:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787698239.123264:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134560640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698239.123269:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134560000:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698239.123283:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134560128:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698239.123283:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698239.123292:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698239.123294:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698239.123299:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6140 00000100:00080000:2.0:1787698239.123304:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698239.123309:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698239.123310:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698239.123318:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698239.123327:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698239/real 1787698239] req@ffff9d85babd4a80 x1874531134560128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698239.123333:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134560768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698239.123350:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134560128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698239.123359:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698239.123369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698239.123374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134560128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698239.123396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134560256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698239.123416:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698239.123425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698239/real 1787698239] req@ffff9d85babd4700 x1874531134560256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698239.123433:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134560256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698239.123437:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698239.123439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698239.123441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134560256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698239.123444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134560384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698239.123454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698239.123459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698239/real 1787698239] req@ffff9d85babd5f80 x1874531134560384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698239.123465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134560384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698239.123470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698239.123471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698239.123473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134560384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698239.123476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134560512:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698239.123484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698239.123488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698239/real 1787698239] req@ffff9d85babd7b80 x1874531134560512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698239.123494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134560512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698239.123534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698239.123536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698239.123538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134560512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698239.123542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134560640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698239.123552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698239.123558:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698239/real 1787698239] req@ffff9d85babd4e00 x1874531134560640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698239.123565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134560640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698239.123569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698239.123571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698239.123572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134560640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698239.123575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134560768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698239.123583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698239.123588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698239/real 1787698239] req@ffff9d85babd5180 x1874531134560768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698239.123596:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134560768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698239.123603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698239.123605:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698239.123609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134560768:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698239.699077:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698240.723111:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698241.747038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698242.771058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698243.795020:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698244.243056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698244.243069:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698244.243074:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698244.243083:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6146 00000100:00080000:3.0:1787698244.243094:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698244.243100:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698244.243144:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134560896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698244.243181:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698244.243185:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698244.243188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698244.243193:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6146 00000100:00080000:3.0:1787698244.243196:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698244.243200:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698244.243209:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698244.243222:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134561024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698244.243233:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134560896:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698244.243235:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698244.243243:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698244.243246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698244.243250:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6146 00000100:00080000:3.0:1787698244.243253:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698244.243256:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698244.243261:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698244.243271:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134561152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698244.243281:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698244.243284:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698244.243286:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698244.243289:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6146 00000100:00080000:3.0:1787698244.243292:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698244.243295:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698244.243299:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698244.243308:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134561280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698244.243318:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698244.243321:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698244.243323:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698244.243326:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6146 00000100:00080000:3.0:1787698244.243329:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698244.243333:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698244.243337:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698244.243348:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134561408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698244.243357:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698244.243361:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698244.243363:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698244.243367:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6146 00000100:00080000:3.0:1787698244.243370:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698244.243377:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698244.243382:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698244.243383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698244.243394:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134561536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698244.243404:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698244.243408:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698244.243410:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698244.243416:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6146 00000100:00080000:3.0:1787698244.243421:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698244.243426:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698244.243430:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561152:192.168.201.137@tcp:8:kworker.0 00000008:00100000:3.0:1787698244.243440:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698244.243455:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134561664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698244.243475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698244.243503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134561408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698244.243535:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134561536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698244.243569:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134561664:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698244.819032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698245.843040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698246.866991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698247.315112:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698247.315149:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698247.315165:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698247.315202:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698247.315212:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698247.315222:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698247.315233:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698247.315244:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698247.315246:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698244/real 1787698247] req@ffff9d85c0fb2d80 x1874531134561024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698247.315253:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698247.315276:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134561024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698247.315288:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698247.315292:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698247.315296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698247.315303:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698244/real 1787698247] req@ffff9d85c0fb0380 x1874531134561152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698247.315312:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134561152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698247.315323:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698247.315327:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698247.315330:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698247.315335:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698244/real 1787698247] req@ffff9d85c0fb0000 x1874531134561280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698247.315345:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134561280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698247.315351:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698247.315354:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698247.315357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698247.315363:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698244/real 1787698247] req@ffff9d85c0fb1f80 x1874531134561408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698247.315374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134561408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698247.315381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698247.315385:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698247.315387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134561408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698247.315394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698244/real 1787698247] req@ffff9d85c0fb3100 x1874531134560896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698247.315406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134560896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698247.315426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698247.315431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698247.315435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134560896:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698247.315462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698244/real 1787698247] req@ffff9d85c0fb1880 x1874531134561536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698247.315474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134561536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698247.315481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698247.315484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698247.315487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134561536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698247.315492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698244/real 1787698247] req@ffff9d85c0fb1500 x1874531134561664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698247.315502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134561664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698247.315508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698247.315510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698247.315512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134561664:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698247.891020:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698248.914985:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698249.362945:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698249.362957:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698249.362960:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698249.362971:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6151 00000100:00080000:3.0:1787698249.362980:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698249.362987:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698249.363019:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134561792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698249.363048:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698249.363052:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698249.363054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698249.363058:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6151 00000100:00080000:3.0:1787698249.363062:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698249.363066:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698249.363075:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698249.363089:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134561920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698249.363098:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134561792:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698249.363104:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698249.363108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698249.363110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698249.363114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6151 00000100:00080000:3.0:1787698249.363118:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698249.363122:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698249.363128:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698249.363141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134562048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698249.363151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698249.363155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698249.363157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698249.363163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6151 00000800:00000100:1.0:1787698249.363164:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698249.363169:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698249.363175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698249.363180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698249.363192:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134562176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698249.363201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698249.363210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698249.363212:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698249.363218:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6151 00000100:00100000:1.0:1787698249.363219:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698249/real 1787698249] req@ffff9d85c0fb0700 x1874531134561792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698249.363238:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698249.363241:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134561792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698249.363245:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698249.363255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698249.363259:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698249.363266:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787698249.363271:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134562304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698249.363273:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134561792:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698249.363288:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698249.363292:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698249.363294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698249.363299:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6151 00000100:00080000:3.0:1787698249.363303:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698249.363306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561920:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698249.363310:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698249.363318:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698249.363331:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134562432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698249.363340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698249.363347:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698249.363352:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698249.363353:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698249.363360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698249/real 1787698249] req@ffff9d85c0fb3480 x1874531134561920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698249.363360:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6151 00000100:00080000:3.0:1787698249.363365:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698249.363374:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698249.363380:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698249.363381:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134561920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698249.363393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698249.363397:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134562560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698249.363400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698249.363407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134561920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698249.363417:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698249.363445:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698249.363457:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698249/real 1787698249] req@ffff9d85c0fb2300 x1874531134562048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698249.363468:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134562048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698249.363479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698249.363482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698249.363485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698249.363490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698249.363507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698249.363516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698249/real 1787698249] req@ffff9d85c0fb1500 x1874531134562176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698249.363525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134562176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698249.363532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698249.363534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698249.363537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698249.363542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134562304:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698249.363560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698249.363570:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698249/real 1787698249] req@ffff9d85c0fb1880 x1874531134562304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698249.363586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134562304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698249.363592:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698249.363595:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698249.363598:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134562304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698249.363614:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134562432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698249.363632:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698249.363642:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698249/real 1787698249] req@ffff9d85c0fb3100 x1874531134562432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698249.363652:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134562432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698249.363658:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698249.363661:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698249.363664:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134562432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698249.363669:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134562560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698249.363686:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698249.363697:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698249/real 1787698249] req@ffff9d85c0fb1f80 x1874531134562560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698249.363707:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134562560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698249.363715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698249.363748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698249.363752:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134562560:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698249.939025:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698250.962957:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698251.987029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698253.010999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698254.035017:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698254.482952:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698254.482960:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698254.482963:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698254.482970:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6156 00000100:00080000:3.0:1787698254.482980:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698254.482986:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698254.483014:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134562688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698254.483036:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698254.483039:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698254.483040:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698254.483043:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6156 00000100:00080000:3.0:1787698254.483045:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698254.483047:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698254.483053:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698254.483060:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134562816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698254.483067:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698254.483069:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698254.483070:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698254.483073:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6156 00000100:00080000:3.0:1787698254.483075:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698254.483079:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698254.483080:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134562688:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698254.483086:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698254.483097:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134562944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698254.483106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698254.483109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698254.483111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698254.483115:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6156 00000100:00080000:3.0:1787698254.483128:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698254.483136:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698254.483141:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698254.483143:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698254.483157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134563072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698254.483167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698254.483170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698254.483172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698254.483176:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6156 00000100:00080000:3.0:1787698254.483179:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698254.483183:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698254.483188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698254.483191:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698254/real 1787698254] req@ffff9d85c0fb0000 x1874531134562688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698254.483201:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134563200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698254.483212:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698254.483214:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134562688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698254.483218:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698254.483222:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698254.483228:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698254.483229:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6156 00000100:00080000:1.0:1787698254.483233:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698254.483236:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698254.483237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134562688:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698254.483245:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698254.483247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562816:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787698254.483254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698254.483269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134563328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698254.483271:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698254.483281:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698254.483282:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698254/real 1787698254] req@ffff9d85c0fb0380 x1874531134562816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698254.483289:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698254.483294:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698254.483302:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6156 00000100:00100000:1.0:1787698254.483305:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134562816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698254.483307:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698254.483315:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698254.483323:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698254.483324:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698254.483331:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698254.483336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698254.483339:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134563456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698254.483362:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698254.483383:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698254.483391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698254/real 1787698254] req@ffff9d85c0fb2d80 x1874531134562944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698254.483398:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134562944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698254.483407:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698254.483409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698254.483411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134562944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698254.483414:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698254.483424:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698254.483429:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698254/real 1787698254] req@ffff9d85c0fb3800 x1874531134563072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698254.483435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134563072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698254.483440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698254.483441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698254.483443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698254.483446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134563200:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698254.483454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698254.483460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698254/real 1787698254] req@ffff9d85c0fb0a80 x1874531134563200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698254.483466:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134563200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698254.483470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698254.483472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698254.483483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134563200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698254.483489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134563328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698254.483505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698254.483515:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698254/real 1787698254] req@ffff9d85c0fb3b80 x1874531134563328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698254.483526:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134563328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698254.483533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698254.483536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698254.483538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134563328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698254.483542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134563456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698254.483554:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698254.483560:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698254/real 1787698254] req@ffff9d85c0fb1c00 x1874531134563456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698254.483566:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134563456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698254.483571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698254.483572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698254.483574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134563456:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698255.059032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698256.083116:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698257.107001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698258.131022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698259.154991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698259.602907:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698259.602915:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698259.602919:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698259.602927:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6161 00000100:00080000:3.0:1787698259.602935:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698259.602941:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698259.602970:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134563584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698259.602996:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698259.602999:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698259.603001:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698259.603005:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6161 00000100:00080000:3.0:1787698259.603009:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698259.603013:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698259.603022:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698259.603034:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134563584:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698259.603039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134563712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698259.603051:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698259.603055:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698259.603057:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698259.603061:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6161 00000100:00080000:3.0:1787698259.603064:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698259.603069:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698259.603074:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698259.603085:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134563840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698259.603088:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698259.603104:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698259.603107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698259.603108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698259.603113:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6161 00000100:00080000:3.0:1787698259.603116:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698259.603121:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698259.603126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698259.603127:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698259/real 1787698259] req@ffff9d85c0fb0e00 x1874531134563584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698259.603140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134563968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698259.603149:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134563584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698259.603152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698259.603160:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698259.603164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698259.603168:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698259.603169:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6161 00000100:00080000:3.0:1787698259.603176:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698259.603178:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698259.603183:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698259.603186:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134563584:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698259.603190:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698259.603205:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134564096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698259.603210:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563712:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698259.603224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698259.603228:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698259.603230:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698259.603235:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6161 00000100:00080000:3.0:1787698259.603240:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698259.603241:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698259.603246:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698259.603252:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698259.603259:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698259/real 1787698259] req@ffff9d85c0fb2680 x1874531134563712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698259.603265:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134564224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698259.603283:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698259.603285:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134563712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698259.603287:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698259.603290:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698259.603297:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6161 00000100:00080000:3.0:1787698259.603304:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698259.603305:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698259.603310:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698259.603313:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787698259.603317:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698259.603318:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698259.603327:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698259.603332:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134564352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698259.603346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698259.603355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698259/real 1787698259] req@ffff9d85c0fb2a00 x1874531134563840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698259.603363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134563840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698259.603372:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698259.603374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698259.603376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698259.603379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698259.603388:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698259.603393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698259/real 1787698259] req@ffff9d85c0fb1c00 x1874531134563968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698259.603400:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134563968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698259.603404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698259.603406:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698259.603407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134563968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698259.603416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564096:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698259.603432:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698259.603438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698259/real 1787698259] req@ffff9d85c0fb3b80 x1874531134564096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698259.603444:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134564096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698259.603448:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698259.603450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698259.603452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698259.603455:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698259.603464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698259.603469:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698259/real 1787698259] req@ffff9d85c0fb0a80 x1874531134564224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698259.603475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134564224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698259.603479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698259.603481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698259.603482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698259.603485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698259.603492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698259.603497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698259/real 1787698259] req@ffff9d85c0fb3800 x1874531134564352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698259.603508:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134564352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698259.603512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698259.603514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698259.603515:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564352:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698260.179031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698261.202993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698262.227016:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698263.250981:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698264.275009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698264.722910:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698264.722920:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698264.722923:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698264.722933:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6166 00000100:00080000:3.0:1787698264.722941:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698264.722946:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698264.722985:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134564480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698264.723008:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698264.723011:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698264.723013:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698264.723017:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6166 00000100:00080000:3.0:1787698264.723020:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698264.723024:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698264.723033:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698264.723044:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134564480:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698264.723053:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134564608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698264.723064:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698264.723067:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698264.723068:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698264.723071:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6166 00000100:00080000:3.0:1787698264.723074:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698264.723077:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698264.723081:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698264.723089:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134564736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698264.723096:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698264.723096:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698264.723099:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698264.723100:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698264.723103:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6166 00000100:00080000:3.0:1787698264.723105:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698264.723108:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698264.723118:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698264.723130:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134564864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698264.723141:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698264/real 1787698264] req@ffff9d85c0fb2d80 x1874531134564480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698264.723142:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698264.723152:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698264.723154:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698264.723159:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134564480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698264.723159:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6166 00000100:00080000:3.0:1787698264.723162:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698264.723166:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698264.723168:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698264.723170:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698264.723172:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698264.723176:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134564480:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698264.723179:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134564992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698264.723186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698264.723188:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698264.723190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698264.723192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6166 00000100:00100000:1.0:1787698264.723195:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134564608:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698264.723195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698264.723199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698264.723202:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698264.723210:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134565120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698264.723213:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698264.723223:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698264.723224:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698264/real 1787698264] req@ffff9d85c0fb0380 x1874531134564608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698264.723227:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698264.723228:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698264.723231:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6166 00000100:00080000:3.0:1787698264.723234:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698264.723235:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134564608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698264.723237:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698264.723242:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698264.723243:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698264.723246:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698264.723249:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134564608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698264.723250:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134565248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698264.723254:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134564736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698264.723271:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698264.723282:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698264/real 1787698264] req@ffff9d85c0fb0000 x1874531134564736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698264.723291:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134564736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698264.723308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698264.723311:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698264.723314:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134564736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698264.723319:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134564864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698264.723338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698264.723349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698264/real 1787698264] req@ffff9d85c0fb1f80 x1874531134564864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698264.723359:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134564864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698264.723364:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698264.723366:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698264.723368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134564864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698264.723372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564992:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698264.723384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698264.723390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698264/real 1787698264] req@ffff9d85c0fb3100 x1874531134564992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698264.723396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134564992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698264.723400:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698264.723402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698264.723404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134564992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698264.723416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134565120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698264.723431:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698264.723441:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698264/real 1787698264] req@ffff9d85c0fb1880 x1874531134565120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698264.723453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134565120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698264.723485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698264.723489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698264.723492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134565120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698264.723498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134565248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698264.723516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698264.723527:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698264/real 1787698264] req@ffff9d85c0fb1500 x1874531134565248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698264.723535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134565248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698264.723539:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698264.723541:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698264.723543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134565248:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698265.299030:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698266.323010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698267.347009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698268.371023:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698269.395031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698269.842947:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698269.842955:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698269.842958:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698269.842966:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6171 00000100:00080000:3.0:1787698269.842975:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698269.842980:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698269.843009:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134565376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698269.843029:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698269.843031:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698269.843032:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698269.843035:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6171 00000100:00080000:3.0:1787698269.843037:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698269.843040:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698269.843052:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698269.843061:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134565504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698269.843067:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698269.843070:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698269.843071:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698269.843073:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6171 00000100:00080000:3.0:1787698269.843075:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698269.843079:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134565376:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698269.843080:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698269.843089:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698269.843098:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134565632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698269.843104:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698269.843107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698269.843108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698269.843110:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6171 00000100:00080000:3.0:1787698269.843112:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698269.843115:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698269.843118:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698269.843125:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134565760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698269.843131:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698269.843133:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698269.843134:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698269.843140:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6171 00000800:00000100:1.0:1787698269.843142:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698269.843144:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698269.843148:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698269.843151:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698269.843158:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134565888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698269.843164:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698269.843166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698269.843167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698269.843169:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6171 00000100:00080000:3.0:1787698269.843171:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698269.843174:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698269.843176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698269.843181:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134566016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698269.843187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698269.843188:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698269.843190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698269.843193:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698269/real 1787698269] req@ffff9d85c0fb2300 x1874531134565376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698269.843194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6171 00000100:00080000:3.0:1787698269.843199:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698269.843204:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698269.843210:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698269.843219:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134566144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698269.843224:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134565376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698269.843236:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698269.843240:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698269.843245:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134565376:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698269.843255:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134565504:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698269.843275:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698269.843285:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698269/real 1787698269] req@ffff9d85c0fb3480 x1874531134565504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698269.843296:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134565504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698269.843304:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698269.843307:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698269.843310:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134565504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698269.843330:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134565632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698269.843345:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698269.843353:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698269/real 1787698269] req@ffff9d85c0fb0700 x1874531134565632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698269.843364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134565632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698269.843404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698269.843407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698269.843411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134565632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698269.843416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134565760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698269.843432:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698269.843441:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698269/real 1787698269] req@ffff9d85c0fb1500 x1874531134565760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698269.843452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134565760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698269.843459:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698269.843462:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698269.843465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134565760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698269.843470:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134565888:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698269.843484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698269.843501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698269/real 1787698269] req@ffff9d85c0fb1880 x1874531134565888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698269.843513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134565888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698269.843520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698269.843523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698269.843526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134565888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698269.843531:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698269.843546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698269.843554:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698269/real 1787698269] req@ffff9d85c0fb3100 x1874531134566016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698269.843564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134566016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698269.843572:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698269.843575:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698269.843578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698269.843583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698269.843596:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698269.843603:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698269/real 1787698269] req@ffff9d85c0fb1f80 x1874531134566144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698269.843614:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134566144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698269.843621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698269.843623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698269.843626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566144:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698270.419067:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698271.442986:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698272.466978:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698273.491035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698274.515030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698274.962985:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698274.962996:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698274.962999:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698274.963010:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6176 00000100:00080000:2.0:1787698274.963021:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698274.963028:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698274.963078:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134566272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698274.963103:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698274.963107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698274.963109:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698274.963113:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6176 00000100:00080000:2.0:1787698274.963116:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698274.963120:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698274.963129:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698274.963142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134566400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698274.963159:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134566272:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698274.963160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698274.963164:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698274.963166:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698274.963175:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6176 00000100:00080000:2.0:1787698274.963179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698274.963183:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698274.963189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698274.963203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134566528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698274.963212:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698274.963215:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698274.963217:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698274.963221:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6176 00000100:00080000:2.0:1787698274.963224:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698274.963230:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698274.963231:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698274.963237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698274.963251:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134566656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698274.963260:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698274.963263:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698274.963265:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698274.963268:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6176 00000100:00080000:2.0:1787698274.963272:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698274.963277:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698274.963296:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698274.963298:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698274/real 1787698274] req@ffff9d85babd7480 x1874531134566272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698274.963311:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134566784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698274.963321:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134566272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698274.963323:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698274.963333:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698274.963336:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698274.963340:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698274.963344:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6176 00000100:00080000:1.0:1787698274.963347:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698274.963352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134566272:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698274.963353:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698274.963362:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698274.963368:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698274.963381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134566400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698274.963390:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134566912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698274.963401:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698274.963404:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698274.963406:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698274.963409:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6176 00000100:00080000:2.0:1787698274.963414:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698274.963417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698274.963422:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698274.963428:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698274.963430:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698274/real 1787698274] req@ffff9d85babd5c00 x1874531134566400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698274.963441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134566400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698274.963446:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134567040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698274.963448:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698274.963452:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698274.963454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134566400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698274.963459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134566528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698274.963475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698274.963484:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698274/real 1787698274] req@ffff9d85babd4000 x1874531134566528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698274.963493:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134566528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698274.963503:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698274.963505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698274.963507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134566528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698274.963511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134566656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698274.963520:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698274.963525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698274/real 1787698274] req@ffff9d85babd4380 x1874531134566656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698274.963531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134566656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698274.963536:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698274.963538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698274.963539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134566656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698274.963542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566784:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698274.963551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698274.963556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698274/real 1787698274] req@ffff9d85babd6300 x1874531134566784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698274.963562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134566784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698274.963566:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698274.963568:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698274.963570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566784:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698274.963580:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698274.963588:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698274.963594:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698274/real 1787698274] req@ffff9d85babd6a00 x1874531134566912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698274.963600:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134566912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698274.963604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698274.963605:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698274.963607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134566912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698274.963615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698274.963624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698274.963629:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698274/real 1787698274] req@ffff9d85babd6680 x1874531134567040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698274.963636:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134567040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698274.963640:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698274.963642:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698274.963643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567040:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698275.539013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698276.562970:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698277.587013:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698278.610996:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698279.635017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698280.082984:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698280.082994:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698280.082997:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698280.083008:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6181 00000100:00080000:3.0:1787698280.083018:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698280.083024:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698280.083058:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134567168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698280.083081:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698280.083085:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698280.083086:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698280.083090:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6181 00000100:00080000:3.0:1787698280.083093:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698280.083097:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698280.083106:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698280.083117:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134567296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698280.083127:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698280.083131:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134567168:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698280.083132:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698280.083135:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698280.083143:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6181 00000100:00080000:3.0:1787698280.083154:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698280.083162:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698280.083168:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698280.083178:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134567424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698280.083187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698280.083203:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698280.083206:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698280.083206:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698280.083211:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6181 00000100:00080000:3.0:1787698280.083214:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698280.083217:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698280.083221:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698280.083229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134567552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698280.083235:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698280.083237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698280.083238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698280.083241:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6181 00000100:00080000:3.0:1787698280.083242:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698280.083244:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698280.083247:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698280.083253:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134567680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698280.083259:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698280.083262:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698280/real 1787698280] req@ffff9d85c0fb0000 x1874531134567168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698280.083262:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698280.083265:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698280.083270:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6181 00000100:00080000:3.0:1787698280.083274:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698280.083280:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698280.083283:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698280.083289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134567808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698280.083293:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134567168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698280.083297:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698280.083304:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698280.083307:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698280.083314:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6181 00000100:00080000:3.0:1787698280.083316:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698280.083318:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698280.083319:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698280.083325:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698280.083328:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787698280.083332:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134567936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698280.083336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134567168:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698280.083364:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134567296:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698280.083387:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698280.083399:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698280/real 1787698280] req@ffff9d85c0fb0380 x1874531134567296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698280.083411:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134567296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698280.083418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698280.083421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698280.083424:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134567296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698280.083429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134567424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698280.083447:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698280.083457:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698280/real 1787698280] req@ffff9d85c0fb2d80 x1874531134567424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698280.083467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134567424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698280.083480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698280.083483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698280.083486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134567424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698280.083492:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134567552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698280.083508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698280.083518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698280/real 1787698280] req@ffff9d85c0fb3800 x1874531134567552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698280.083530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134567552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698280.083537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698280.083540:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698280.083542:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134567552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698280.083547:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567680:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698280.083564:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698280.083574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698280/real 1787698280] req@ffff9d85c0fb0a80 x1874531134567680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698280.083584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134567680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698280.083591:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698280.083594:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698280.083598:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698280.083603:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698280.083620:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698280.083630:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698280/real 1787698280] req@ffff9d85c0fb3b80 x1874531134567808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698280.083641:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134567808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698280.083648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698280.083651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698280.083654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698280.083659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698280.083676:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698280.083686:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698280/real 1787698280] req@ffff9d85c0fb1c00 x1874531134567936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698280.083696:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134567936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698280.083704:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698280.083708:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698280.083711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134567936:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698280.658997:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698281.683009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698282.707000:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698283.730997:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698284.755000:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698285.202952:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698285.202960:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698285.202963:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698285.202971:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6186 00000100:00080000:3.0:1787698285.202980:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698285.202985:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698285.203015:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134568064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698285.203036:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698285.203039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698285.203040:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698285.203042:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6186 00000100:00080000:3.0:1787698285.203044:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698285.203046:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698285.203053:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698285.203060:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134568192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698285.203067:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698285.203069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698285.203070:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698285.203073:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6186 00000100:00080000:3.0:1787698285.203074:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698285.203078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134568064:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698285.203079:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698285.203085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698285.203091:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134568320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698285.203101:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698285.203114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698285.203117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698285.203120:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6186 00000100:00080000:3.0:1787698285.203124:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698285.203128:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698285.203132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698285.203143:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134568448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698285.203150:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698285.203151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698285.203155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698285.203157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698285.203161:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6186 00000100:00080000:3.0:1787698285.203165:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698285.203167:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698285.203169:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698285.203177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134568576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698285.203182:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698285.203184:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698285.203186:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698285.203188:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6186 00000100:00080000:3.0:1787698285.203191:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698285.203195:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698285/real 1787698285] req@ffff9d85c0fb2a00 x1874531134568064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698285.203195:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698285.203201:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698285.203211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134568704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698285.203216:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134568064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698285.203221:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698285.203229:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698285.203232:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698285.203234:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698285.203236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6186 00000100:00080000:1.0:1787698285.203240:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698285.203240:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698285.203247:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698285.203249:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134568064:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698285.203251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698285.203256:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134568192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698285.203261:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134568832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698285.203276:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698285.203287:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698285/real 1787698285] req@ffff9d85c0fb2680 x1874531134568192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698285.203299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134568192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698285.203307:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698285.203309:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698285.203313:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134568192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698285.203333:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134568320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698285.203347:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698285.203354:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698285/real 1787698285] req@ffff9d85c0fb0e00 x1874531134568320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698285.203361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134568320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698285.203370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698285.203372:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698285.203374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134568320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698285.203377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134568448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698285.203385:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698285.203390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698285/real 1787698285] req@ffff9d85c0fb1c00 x1874531134568448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698285.203396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134568448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698285.203401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698285.203402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698285.203404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134568448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698285.203407:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134568576:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698285.203416:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698285.203421:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698285/real 1787698285] req@ffff9d85c0fb3b80 x1874531134568576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698285.203427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134568576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698285.203431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698285.203433:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698285.203435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134568576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698285.203438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134568704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698285.203446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698285.203450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698285/real 1787698285] req@ffff9d85c0fb0a80 x1874531134568704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698285.203456:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134568704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698285.203461:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698285.203462:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698285.203464:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134568704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698285.203467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134568832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698285.203475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698285.203479:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698285/real 1787698285] req@ffff9d85c0fb3800 x1874531134568832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698285.203484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134568832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698285.203491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698285.203493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698285.203497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134568832:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698285.779001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698286.802959:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698287.826979:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698288.851021:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698289.875011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698290.322918:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698290.322927:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698290.322930:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698290.322941:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6192 00000100:00080000:3.0:1787698290.322953:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698290.322961:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698290.323005:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134568960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698290.323043:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698290.323045:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698290.323047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698290.323051:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6192 00000100:00080000:3.0:1787698290.323054:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698290.323057:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698290.323068:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698290.323082:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134569088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698290.323085:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134568960:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698290.323109:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698290.323112:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698290.323113:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698290.323117:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6192 00000100:00080000:3.0:1787698290.323119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698290.323122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698290.323126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698290.323137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134569216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698290.323142:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698290.323145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698290.323147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698290.323150:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6192 00000100:00080000:3.0:1787698290.323153:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698290.323156:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698290.323160:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698290.323167:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698290.323169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134569344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698290.323177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698290.323179:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698290.323182:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698290.323185:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6192 00000100:00080000:3.0:1787698290.323189:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698290.323193:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698290.323196:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698290.323202:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134569472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698290.323207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698290.323209:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698290.323210:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698290.323214:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6192 00000100:00080000:3.0:1787698290.323218:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698290.323219:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698290/real 1787698290] req@ffff9d85c0fb2d80 x1874531134568960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698290.323226:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698290.323239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698290.323251:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134569600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698290.323252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134568960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698290.323312:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698290.323314:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698290.323316:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698290.323317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698290.323322:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6192 00000100:00080000:1.0:1787698290.323326:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698290.323327:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698290.323335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134568960:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698290.323335:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698290.323348:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698290.323366:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134569728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698290.323371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698290.323405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698290.323415:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698290/real 1787698290] req@ffff9d85c0fb0380 x1874531134569088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698290.323423:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134569088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698290.323429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698290.323431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698290.323433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698290.323437:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698290.323448:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698290.323453:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698290/real 1787698290] req@ffff9d85c0fb0000 x1874531134569216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698290.323460:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134569216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698290.323468:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698290.323470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698290.323473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698290.323479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698290.323505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698290.323516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698290/real 1787698290] req@ffff9d85c0fb1f80 x1874531134569344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698290.323524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134569344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698290.323530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698290.323532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698290.323534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698290.323537:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134569472:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698290.323550:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698290.323563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698290/real 1787698290] req@ffff9d85c0fb3100 x1874531134569472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698290.323570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134569472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698290.323575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698290.323577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698290.323579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134569472:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698290.323583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134569600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698290.323592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698290.323598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698290/real 1787698290] req@ffff9d85c0fb1880 x1874531134569600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698290.323605:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134569600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698290.323613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698290.323616:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698290.323619:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134569600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698290.323636:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134569728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698290.323654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698290.323661:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698290/real 1787698290] req@ffff9d85c0fb1500 x1874531134569728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698290.323668:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134569728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698290.323672:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698290.323675:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698290.323676:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134569728:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698290.899081:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698291.923087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698292.947084:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698293.971037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698294.994989:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698295.442975:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698295.442985:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698295.442988:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698295.442997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6197 00000100:00080000:2.0:1787698295.443015:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698295.443022:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698295.443055:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134569856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698295.443078:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698295.443082:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698295.443083:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698295.443088:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6197 00000100:00080000:2.0:1787698295.443092:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698295.443095:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698295.443103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698295.443116:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134569984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698295.443121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134569856:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698295.443133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698295.443138:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698295.443140:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698295.443145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6197 00000100:00080000:2.0:1787698295.443148:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698295.443152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698295.443157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698295.443168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134570112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698295.443176:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698295.443181:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698295.443185:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698295.443187:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698295.443192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6197 00000100:00080000:2.0:1787698295.443195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698295.443199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698295.443203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698295.443210:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698295/real 1787698295] req@ffff9d85babd7800 x1874531134569856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698295.443218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134570240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698295.443226:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134569856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698295.443256:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698295.443257:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698295.443263:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698295.443264:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787698295.443268:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134569856:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698295.443269:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698295.443277:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6197 00000100:00080000:2.0:1787698295.443281:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698295.443285:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698295.443288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569984:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787698295.443290:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698295.443307:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698295.443314:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134570368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698295.443317:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698295/real 1787698295] req@ffff9d85babd6d80 x1874531134569984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698295.443343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134569984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698295.443344:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698295.443352:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698295.443356:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698295.443357:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698295.443364:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698295.443364:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6197 00000100:00100000:1.0:1787698295.443372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134569984:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698295.443373:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698295.443380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134570112:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698295.443381:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698295.443388:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698295.443396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698295.443403:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134570496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698295.443408:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698295/real 1787698295] req@ffff9d85babd5500 x1874531134570112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698295.443421:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698295.443426:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698295.443428:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698295.443432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134570112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698295.443434:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6197 00000100:00080000:2.0:1787698295.443440:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698295.443443:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698295.443449:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698295.443462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698295.443462:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134570624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698295.443470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698295.443478:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134570112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698295.443486:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134570240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698295.443507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698295.443516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698295/real 1787698295] req@ffff9d85babd6680 x1874531134570240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698295.443523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134570240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698295.443528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698295.443530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698295.443531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134570240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698295.443540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134570368:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698295.443552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698295.443557:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698295/real 1787698295] req@ffff9d85babd6a00 x1874531134570368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698295.443564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134570368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698295.443568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698295.443569:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698295.443572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134570368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698295.443575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134570496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698295.443583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698295.443588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698295/real 1787698295] req@ffff9d85babd6300 x1874531134570496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698295.443594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134570496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698295.443598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698295.443599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698295.443601:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134570496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698295.443603:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134570624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698295.443611:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698295.443616:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698295/real 1787698295] req@ffff9d85babd4380 x1874531134570624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698295.443622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134570624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698295.443626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698295.443628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698295.443629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134570624:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698296.019064:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698297.043012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698298.066983:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698299.090992:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698300.114963:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698300.562933:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698300.562943:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698300.562947:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698300.562958:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6202 00000100:00080000:3.0:1787698300.562967:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698300.562974:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698300.563007:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134570752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698300.563029:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698300.563031:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698300.563032:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698300.563035:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6202 00000100:00080000:3.0:1787698300.563037:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698300.563040:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698300.563047:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698300.563054:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134570880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698300.563060:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698300.563063:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698300.563064:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698300.563066:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6202 00000100:00080000:3.0:1787698300.563068:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698300.563070:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698300.563073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698300.563078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134570752:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698300.563080:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134571008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698300.563090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698300.563092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698300.563093:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698300.563095:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6202 00000100:00080000:3.0:1787698300.563097:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698300.563099:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698300.563102:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698300.563110:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134571136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698300.563115:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698300.563117:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698300.563118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698300.563120:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6202 00000100:00080000:3.0:1787698300.563123:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698300.563125:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698300.563127:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698300.563133:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698300.563135:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134571264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698300.563141:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698300.563143:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698300.563144:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698300.563146:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6202 00000100:00080000:3.0:1787698300.563149:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698300.563151:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698300.563154:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698300.563160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134571392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698300.563168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698300.563171:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698300.563172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698300.563175:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6202 00000100:00100000:1.0:1787698300.563178:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698300/real 1787698300] req@ffff9d85c0fb0700 x1874531134570752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698300.563179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698300.563187:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698300.563196:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698300.563206:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134570752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698300.563210:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134571520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698300.563229:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698300.563235:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698300.563239:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134570752:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698300.563247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134570880:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698300.563268:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698300.563279:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698300/real 1787698300] req@ffff9d85c0fb3480 x1874531134570880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698300.563302:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134570880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698300.563308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698300.563311:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698300.563314:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134570880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698300.563334:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698300.563353:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698300.563362:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698300/real 1787698300] req@ffff9d85c0fb2300 x1874531134571008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698300.563373:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134571008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698300.563387:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698300.563389:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698300.563393:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698300.563398:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698300.563417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698300.563427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698300/real 1787698300] req@ffff9d85c0fb1500 x1874531134571136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698300.563434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134571136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698300.563439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698300.563441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698300.563443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698300.563446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134571264:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698300.563458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698300.563463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698300/real 1787698300] req@ffff9d85c0fb1880 x1874531134571264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698300.563469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134571264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698300.563473:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698300.563475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698300.563477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134571264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698300.563480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134571392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698300.563488:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698300.563493:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698300/real 1787698300] req@ffff9d85c0fb3100 x1874531134571392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698300.563499:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134571392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698300.563503:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698300.563504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698300.563506:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134571392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698300.563508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134571520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698300.563516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698300.563521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698300/real 1787698300] req@ffff9d85c0fb1f80 x1874531134571520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698300.563527:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134571520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698300.563532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698300.563533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698300.563535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134571520:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698301.139029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698302.162971:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698303.186979:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698304.211016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698305.235024:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698305.683001:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698305.683019:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698305.683024:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698305.683036:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6207 00000100:00080000:2.0:1787698305.683050:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698305.683058:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698305.683105:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134571648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698305.683138:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698305.683142:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698305.683144:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698305.683148:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6207 00000100:00080000:2.0:1787698305.683152:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698305.683156:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698305.683167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698305.683180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134571776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698305.683190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698305.683193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698305.683195:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698305.683200:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6207 00000100:00080000:2.0:1787698305.683203:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698305.683208:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698305.683214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698305.683228:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134571648:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698305.683228:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134571904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698305.683263:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698305.683269:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698305.683271:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698305.683276:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6207 00000100:00080000:2.0:1787698305.683280:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698305.683286:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698305.683292:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698305.683310:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134572032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698305.683312:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698305.683342:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698305.683344:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698305.683345:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698305.683349:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6207 00000100:00080000:2.0:1787698305.683354:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698305.683360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698305/real 1787698305] req@ffff9d85babd4000 x1874531134571648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698305.683364:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698305.683378:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698305.683393:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134571648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698305.683399:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134572160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698305.683419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698305.683426:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698305.683427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698305.683438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134571648:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698305.683439:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698305.683450:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571776:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698305.683455:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698305.683464:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6207 00000100:00080000:2.0:1787698305.683469:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698305.683475:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698305.683484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698305.683487:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698305.683499:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698305/real 1787698305] req@ffff9d85babd5c00 x1874531134571776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698305.683507:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134572288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698305.683513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134571776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698305.683538:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698305.683542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698305.683544:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698305.683548:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698305.683554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698305.683559:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6207 00000100:00100000:1.0:1787698305.683567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571776:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698305.683569:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698305.683576:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698305.683581:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698305.683594:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134572416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698305.683605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698305.683629:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698305.683639:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698305/real 1787698305] req@ffff9d85babd7480 x1874531134571904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698305.683646:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134571904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698305.683655:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698305.683657:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698305.683659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134571904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698305.683663:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698305.683673:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698305.683678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698305/real 1787698305] req@ffff9d85babd5180 x1874531134572032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698305.683684:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134572032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698305.683689:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698305.683690:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698305.683692:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698305.683695:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134572160:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698305.683704:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698305.683709:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698305/real 1787698305] req@ffff9d85babd4e00 x1874531134572160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698305.683716:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134572160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698305.683724:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698305.683727:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698305.683730:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134572160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698305.683741:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134572288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698305.683788:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698305.683801:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698305/real 1787698305] req@ffff9d85babd7b80 x1874531134572288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698305.683812:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134572288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698305.683817:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698305.683819:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698305.683821:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134572288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698305.683832:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134572416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698305.683845:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698305.683897:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698305/real 1787698305] req@ffff9d85babd5f80 x1874531134572416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698305.683910:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134572416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698305.683918:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698305.683921:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698305.683924:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134572416:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698306.259051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698307.282995:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698308.307018:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698309.331002:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698310.355006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698310.802948:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698310.802958:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698310.802962:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698310.802973:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6212 00000100:00080000:3.0:1787698310.802983:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698310.802990:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698310.803022:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134572544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698310.803055:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698310.803058:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698310.803060:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698310.803064:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6212 00000100:00080000:3.0:1787698310.803067:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698310.803071:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698310.803079:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698310.803091:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134572672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698310.803101:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698310.803103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134572544:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698310.803107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698310.803111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698310.803116:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6212 00000100:00080000:3.0:1787698310.803119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698310.803122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698310.803127:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698310.803139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134572800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698310.803148:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698310.803151:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698310.803153:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698310.803157:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6212 00000100:00080000:3.0:1787698310.803161:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698310.803165:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698310.803170:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698310.803191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134572928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698310.803201:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698310.803205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698310.803206:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698310.803210:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6212 00000100:00080000:3.0:1787698310.803214:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698310.803218:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698310.803223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698310.803225:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698310.803240:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134573056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698310.803255:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698310.803258:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698310.803260:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698310.803265:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572800:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698310.803268:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6212 00000100:00080000:3.0:1787698310.803274:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698310.803279:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698310.803284:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698310.803299:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698310.803300:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134573184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698310.803313:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698310.803317:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698310.803319:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698310.803323:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6212 00000100:00080000:3.0:1787698310.803327:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698310.803328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573056:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698310.803335:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698310.803342:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698310.803357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698310.803358:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134573312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698310.803401:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573312:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698311.379000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698312.403026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698313.426987:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698313.875083:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698313.875121:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698313.875135:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698313.875173:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698313.875182:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698313.875189:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698313.875196:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698313.875203:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698313.875211:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698313.875254:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698310/real 1787698313] req@ffff9d85c0fb1c00 x1874531134573312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698313.875273:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134573312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698313.875299:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698313.875303:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698313.875307:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698313.875314:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698310/real 1787698313] req@ffff9d85c0fb0a80 x1874531134573056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698313.875325:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134573056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698313.875333:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698313.875335:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698313.875338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698313.875346:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698310/real 1787698313] req@ffff9d85c0fb3b80 x1874531134573184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698313.875356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134573184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698313.875363:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698313.875365:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698313.875368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698313.875373:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698310/real 1787698313] req@ffff9d85c0fb0380 x1874531134572672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698313.875383:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134572672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698313.875389:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698313.875392:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698313.875395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698313.875400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698310/real 1787698313] req@ffff9d85c0fb2d80 x1874531134572800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698313.875408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134572800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698313.875414:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698313.875417:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698313.875420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698313.875425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698310/real 1787698313] req@ffff9d85c0fb3800 x1874531134572928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698313.875434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134572928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698313.875440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698313.875444:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698313.875446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134572928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698313.875451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698310/real 1787698313] req@ffff9d85c0fb0000 x1874531134572544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698313.875461:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134572544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698313.875478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698313.875481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698313.875483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134572544:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787698314.451046:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698315.474963:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698315.922968:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698315.922976:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698315.922980:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698315.922988:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6217 00000100:00080000:3.0:1787698315.922998:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698315.923010:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698315.923041:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134573440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698315.923079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698315.923083:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698315.923085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698315.923089:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6217 00000100:00080000:3.0:1787698315.923093:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698315.923096:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698315.923105:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698315.923118:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134573568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698315.923127:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698315.923130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698315.923131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698315.923134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6217 00000100:00080000:3.0:1787698315.923137:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698315.923140:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698315.923145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698315.923154:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134573696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698315.923167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698315.923170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698315.923171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698315.923174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6217 00000100:00080000:3.0:1787698315.923177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698315.923180:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698315.923185:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698315.923195:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134573824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698315.923203:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698315.923206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698315.923207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698315.923210:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6217 00000100:00080000:3.0:1787698315.923212:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698315.923215:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698315.923219:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698315.923227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134573952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698315.923234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698315.923237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698315.923238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698315.923241:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6217 00000100:00080000:3.0:1787698315.923244:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698315.923247:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698315.923251:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698315.923258:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134574080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698315.923265:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698315.923267:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698315.923269:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698315.923272:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6217 00000100:00080000:3.0:1787698315.923274:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698315.923278:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698315.923281:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698315.923288:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134574208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698315.923374:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134573440:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698315.923436:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698315.923488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698315/real 1787698315] req@ffff9d85c0fb0e00 x1874531134573440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698315.923504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134573440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698315.923512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698315.923516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698315.923520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134573440:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698315.923526:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134573568:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698315.923552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698315.923564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698315/real 1787698315] req@ffff9d85c0fb2680 x1874531134573568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698315.923575:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134573568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698315.923601:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698315.923604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698315.923608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134573568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698315.923613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134573696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698315.923635:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698315.923646:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698315/real 1787698315] req@ffff9d85c0fb2a00 x1874531134573696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698315.923656:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134573696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698315.923663:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698315.923665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698315.923668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134573696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698315.923673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134573824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698315.923690:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698315.923699:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698315/real 1787698315] req@ffff9d85c0fb0000 x1874531134573824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698315.923708:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134573824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698315.923714:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698315.923717:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698315.923720:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134573824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698315.923725:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698315.923739:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698315.923748:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698315/real 1787698315] req@ffff9d85c0fb3800 x1874531134573952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698315.923763:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134573952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698315.923770:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698315.923773:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698315.923776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134573952:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698315.923781:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698315.923797:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698315.923806:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698315/real 1787698315] req@ffff9d85c0fb2d80 x1874531134574080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698315.923821:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134574080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698315.923829:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698315.923831:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698315.923834:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698315.923838:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698315.923878:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698315.923889:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698315/real 1787698315] req@ffff9d85c0fb0380 x1874531134574208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698315.923900:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134574208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698315.923908:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698315.923911:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698315.923914:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574208:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698316.499053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698317.522974:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698318.547028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698319.570975:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698320.594996:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698321.042941:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698321.042949:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698321.042952:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698321.042960:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6222 00000100:00080000:3.0:1787698321.042968:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698321.042973:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698321.043002:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134574336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698321.043025:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698321.043028:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698321.043029:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698321.043032:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6222 00000100:00080000:3.0:1787698321.043035:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698321.043037:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698321.043043:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698321.043050:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134574464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698321.043056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698321.043058:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698321.043059:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698321.043061:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6222 00000100:00080000:3.0:1787698321.043062:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698321.043064:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698321.043067:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698321.043071:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134574592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698321.043075:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698321.043077:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698321.043078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698321.043080:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6222 00000100:00080000:3.0:1787698321.043081:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698321.043083:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698321.043086:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698321.043091:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134574720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698321.043096:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698321.043098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698321.043098:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698321.043100:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6222 00000100:00080000:3.0:1787698321.043102:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698321.043104:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698321.043106:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698321.043110:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134574848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698321.043115:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698321.043116:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698321.043117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698321.043119:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6222 00000100:00080000:3.0:1787698321.043120:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698321.043122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698321.043124:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698321.043129:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134574976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698321.043133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698321.043134:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698321.043136:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698321.043138:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6222 00000100:00080000:3.0:1787698321.043141:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698321.043143:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698321.043145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698321.043150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134575104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698321.043164:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134574336:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698321.043222:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698321.043262:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698321/real 1787698321] req@ffff9d85c0fb3b80 x1874531134574336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698321.043275:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134574336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698321.043287:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698321.043290:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698321.043293:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134574336:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698321.043297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134574464:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698321.043318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698321.043325:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698321/real 1787698321] req@ffff9d85c0fb0a80 x1874531134574464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698321.043332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134574464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698321.043337:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698321.043339:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698321.043341:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134574464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698321.043344:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134574592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698321.043354:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698321.043359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698321/real 1787698321] req@ffff9d85c0fb1c00 x1874531134574592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698321.043365:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134574592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698321.043369:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698321.043371:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698321.043372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134574592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698321.043376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134574720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698321.043385:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698321.043390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698321/real 1787698321] req@ffff9d85c0fb1f80 x1874531134574720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698321.043396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134574720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698321.043401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698321.043402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698321.043404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134574720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698321.043406:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574848:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698321.043415:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698321.043420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698321/real 1787698321] req@ffff9d85c0fb3100 x1874531134574848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698321.043426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134574848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698321.043430:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698321.043432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698321.043433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574848:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698321.043436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698321.043444:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698321.043449:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698321/real 1787698321] req@ffff9d85c0fb1880 x1874531134574976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698321.043455:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134574976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698321.043459:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698321.043461:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698321.043462:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134574976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698321.043481:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134575104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698321.043490:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698321.043525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698321/real 1787698321] req@ffff9d85c0fb1500 x1874531134575104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698321.043531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134575104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698321.043535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698321.043538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698321.043539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134575104:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698321.618983:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698322.643019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698323.669629:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698324.691002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698325.715015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698326.162998:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698326.163009:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698326.163012:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698326.163022:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6227 00000100:00080000:3.0:1787698326.163033:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698326.163039:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698326.163074:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134575232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698326.163102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698326.163106:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698326.163108:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698326.163112:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6227 00000100:00080000:3.0:1787698326.163115:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698326.163118:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698326.163139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698326.163150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134575232:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698326.163154:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134575360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698326.163167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698326.163170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698326.163171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698326.163175:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6227 00000100:00080000:3.0:1787698326.163178:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698326.163181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698326.163186:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698326.163199:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134575488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698326.163208:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698326.163211:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698326.163213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698326.163216:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6227 00000100:00080000:3.0:1787698326.163220:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698326.163225:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698326.163226:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698326.163233:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698326.163248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134575616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698326.163258:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698326.163261:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698326.163262:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698326.163266:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6227 00000100:00080000:3.0:1787698326.163269:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698326.163273:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698326.163277:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698326.163286:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698326.163293:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698326.163296:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698326.163297:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698326.163301:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6227 00000100:00080000:3.0:1787698326.163303:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698326.163307:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698326.163311:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698326.163320:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134575872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698326.163328:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698326.163331:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698326.163332:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698326.163335:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6227 00000100:00080000:3.0:1787698326.163338:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698326.163341:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698326.163345:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698326.163356:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134576000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698326.163362:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698326/real 1787698326] req@ffff9d85c0fb2300 x1874531134575232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698326.163396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134575232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698326.163433:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698326.163437:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698326.163441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134575232:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698326.163462:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134575360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698326.163484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698326.163496:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698326/real 1787698326] req@ffff9d85c0fb3480 x1874531134575360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698326.163508:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134575360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698326.163515:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698326.163518:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698326.163521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134575360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698326.163526:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134575488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698326.163544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698326.163553:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698326/real 1787698326] req@ffff9d85c0fb0700 x1874531134575488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698326.163564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134575488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698326.163571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698326.163575:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698326.163578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134575488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698326.163584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134575616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698326.163601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698326.163611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698326/real 1787698326] req@ffff9d85c0fb1500 x1874531134575616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698326.163621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134575616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698326.163629:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698326.163632:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698326.163635:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134575616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698326.163641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134575744:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698326.163657:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698326.163666:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698326/real 1787698326] req@ffff9d85c0fb1880 x1874531134575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698326.163677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698326.163684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698326.163687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698326.163690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134575744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698326.163696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134575872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698326.163712:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698326.163722:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698326/real 1787698326] req@ffff9d85c0fb3100 x1874531134575872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698326.163733:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134575872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698326.163740:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698326.163743:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698326.163745:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134575872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698326.163751:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698326.163769:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698326.163780:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698326/real 1787698326] req@ffff9d85c0fb1f80 x1874531134576000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698326.163791:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134576000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698326.163798:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698326.163802:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698326.163805:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576000:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698326.739028:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698327.763052:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698328.786991:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698329.810956:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698330.834983:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698331.282982:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698331.282993:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698331.282998:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698331.283018:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6233 00000100:00080000:1.0:1787698331.283028:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698331.283035:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698331.283072:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134576128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698331.283102:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698331.283106:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698331.283107:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698331.283112:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6233 00000100:00080000:1.0:1787698331.283117:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698331.283121:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698331.283130:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698331.283144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134576256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698331.283158:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698331.283160:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134576128:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698331.283163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698331.283170:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698331.283186:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6233 00000100:00080000:1.0:1787698331.283189:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698331.283194:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698331.283201:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698331.283217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134576384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698331.283227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698331.283230:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698331.283231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698331.283235:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6233 00000100:00080000:1.0:1787698331.283241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698331.283245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698331.283250:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698331.283258:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698331.283263:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134576512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698331.283275:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698331.283278:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698331.283280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698331.283284:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6233 00000100:00080000:1.0:1787698331.283288:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698331.283292:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698331.283297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698331.283314:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134576640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698331.283336:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698331/real 1787698331] req@ffff9d85b9d2f480 x1874531134576128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698331.283337:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698331.283352:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698331.283355:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698331.283370:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6233 00000100:00080000:1.0:1787698331.283376:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698331.283380:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698331.283386:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698331.283388:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134576128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698331.283403:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134576768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698331.283405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698331.283415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698331.283428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134576128:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698331.283436:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698331.283440:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698331.283444:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698331.283449:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6233 00000100:00080000:1.0:1787698331.283453:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698331.283456:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698331.283462:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698331.283468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134576256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698331.283478:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134576896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698331.283504:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698331.283521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698331/real 1787698331] req@ffff9d85b9d2ca80 x1874531134576256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698331.283535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134576256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698331.283543:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698331.283546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698331.283550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134576256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698331.283556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134576384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698331.283572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698331.283594:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698331/real 1787698331] req@ffff9d85b9d2c000 x1874531134576384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698331.283607:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134576384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698331.283613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698331.283617:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698331.283620:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134576384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698331.283632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134576512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698331.283650:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698331.283659:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698331/real 1787698331] req@ffff9d85b9d2ce00 x1874531134576512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698331.283670:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134576512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698331.283682:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698331.283685:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698331.283688:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134576512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698331.283694:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576640:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698331.283712:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698331.283722:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698331/real 1787698331] req@ffff9d85b9d2e300 x1874531134576640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698331.283733:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134576640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698331.283744:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698331.283748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698331.283752:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698331.283760:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698331.283779:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698331.283789:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698331/real 1787698331] req@ffff9d85b9d2fb80 x1874531134576768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698331.283800:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134576768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698331.283808:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698331.283811:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698331.283815:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698331.283839:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698331.283910:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698331.283922:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698331/real 1787698331] req@ffff9d85b9d2f800 x1874531134576896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698331.283944:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134576896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698331.283951:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698331.283954:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698331.283956:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134576896:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698331.859040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698332.883011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698333.907003:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698334.931009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698335.955030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698336.402959:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698336.402966:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698336.402969:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698336.402977:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6238 00000100:00080000:1.0:1787698336.402986:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698336.402992:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698336.403021:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134577024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698336.403051:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698336.403053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698336.403057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698336.403060:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6238 00000100:00080000:1.0:1787698336.403062:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698336.403065:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698336.403073:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698336.403088:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134577024:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698336.403089:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134577152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698336.403104:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698336.403108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698336.403110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698336.403114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6238 00000100:00080000:1.0:1787698336.403118:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698336.403121:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698336.403126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698336.403136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134577280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698336.403144:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698336.403152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698336.403156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698336.403158:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698336.403163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6238 00000100:00080000:1.0:1787698336.403168:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698336.403172:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698336.403177:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698336.403189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134577408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698336.403191:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698336/real 1787698336] req@ffff9d85b9d2dc00 x1874531134577024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698336.403206:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698336.403209:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698336.403211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698336.403214:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6238 00000100:00080000:1.0:1787698336.403217:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698336.403219:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134577024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698336.403222:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698336.403228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698336.403250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134577536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698336.403257:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698336.403258:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698336.403264:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698336.403265:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698336.403268:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698336.403271:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6238 00000100:00080000:1.0:1787698336.403274:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698336.403277:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134577024:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698336.403279:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698336.403287:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698336.403294:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134577664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698336.403304:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698336.403307:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698336.403308:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698336.403313:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134577152:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698336.403314:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6238 00000100:00080000:1.0:1787698336.403321:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698336.403326:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698336.403331:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698336.403346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698336.403346:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134577792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698336.403364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698336/real 1787698336] req@ffff9d85b9d2c380 x1874531134577152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698336.403375:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134577152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698336.403390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698336.403393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698336.403395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134577152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698336.403399:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134577280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698336.403417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698336.403427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698336/real 1787698336] req@ffff9d85b9d2c700 x1874531134577280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698336.403438:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134577280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698336.403446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698336.403450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698336.403453:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134577280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698336.403468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134577408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698336.403485:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698336.403492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698336/real 1787698336] req@ffff9d85b9d2ed80 x1874531134577408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698336.403498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134577408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698336.403504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698336.403506:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698336.403508:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134577408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698336.403511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134577536:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698336.403520:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698336.403525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698336/real 1787698336] req@ffff9d85b9d2d500 x1874531134577536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698336.403531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134577536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698336.403535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698336.403537:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698336.403539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134577536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698336.403549:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134577664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698336.403562:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698336.403572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698336/real 1787698336] req@ffff9d85b9d2e680 x1874531134577664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698336.403582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134577664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698336.403589:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698336.403592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698336.403595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134577664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698336.403610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134577792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698336.403634:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698336.403654:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698336/real 1787698336] req@ffff9d85b9d2dc00 x1874531134577792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698336.403665:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134577792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698336.403670:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698336.403672:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698336.403675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134577792:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698336.978999:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698338.003028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698339.027018:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698340.051015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698341.074972:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698341.522967:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698341.522977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698341.522981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698341.522998:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6243 00000100:00080000:0.0:1787698341.523007:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698341.523013:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698341.523042:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134577920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698341.523063:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698341.523066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698341.523067:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698341.523070:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6243 00000100:00080000:0.0:1787698341.523072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698341.523075:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698341.523083:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698341.523090:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134577920:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698341.523093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134578048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698341.523101:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698341.523103:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698341.523104:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698341.523106:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6243 00000100:00080000:0.0:1787698341.523108:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698341.523111:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698341.523114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698341.523121:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134578176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698341.523127:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698341.523129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698341.523130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698341.523132:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6243 00000100:00080000:0.0:1787698341.523160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698341.523164:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698341.523168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698341.523169:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698341.523184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134578304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698341.523194:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698341.523197:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698341.523198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698341.523201:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6243 00000100:00080000:0.0:1787698341.523204:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698341.523209:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698341.523213:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698341.523220:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698341/real 1787698341] req@ffff9d85b9d0b480 x1874531134577920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698341.523227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134578432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698341.523244:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698341.523247:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698341.523250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698341.523250:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134577920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698341.523258:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6243 00000100:00080000:1.0:1787698341.523267:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698341.523270:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698341.523273:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698341.523279:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698341.523279:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134577920:192.168.201.137@tcp:250:kworker.0 00000008:00100000:0.0:1787698341.523288:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698341.523303:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134578560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698341.523310:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578048:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698341.523316:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698341.523321:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698341.523322:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698341.523327:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6243 00000100:00080000:0.0:1787698341.523335:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698341.523339:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698341.523346:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698341.523354:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698341.523367:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134578688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698341.523369:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698341/real 1787698341] req@ffff9d85b9d0a680 x1874531134578048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698341.523397:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134578048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698341.523405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698341.523409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698341.523413:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578048:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698341.523420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698341.523442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698341.523454:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698341/real 1787698341] req@ffff9d85b9d0bb80 x1874531134578176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698341.523465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134578176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698341.523472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698341.523484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698341.523488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698341.523494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698341.523513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698341.523522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698341/real 1787698341] req@ffff9d85b9d0ad80 x1874531134578304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698341.523529:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134578304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698341.523534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698341.523536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698341.523537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698341.523541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134578432:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698341.523551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698341.523556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698341/real 1787698341] req@ffff9d85b9d08000 x1874531134578432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698341.523574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134578432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698341.523579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698341.523581:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698341.523583:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134578432:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698341.523594:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134578560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698341.523605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698341.523611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698341/real 1787698341] req@ffff9d85b9d08a80 x1874531134578560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698341.523617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134578560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698341.523621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698341.523623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698341.523625:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134578560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698341.523628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134578688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698341.523636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698341.523640:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698341/real 1787698341] req@ffff9d85b9d08e00 x1874531134578688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698341.523646:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134578688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698341.523654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698341.523655:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698341.523657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134578688:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698342.099040:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698343.122968:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698344.147014:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698345.171003:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698346.194990:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698346.642972:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698346.642982:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698346.642985:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698346.642994:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6248 00000100:00080000:3.0:1787698346.643003:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698346.643010:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698346.643050:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134578816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698346.643098:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698346.643101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698346.643103:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698346.643107:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6248 00000100:00080000:3.0:1787698346.643112:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698346.643116:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698346.643127:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134578816:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698346.643132:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698346.643148:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698346.643158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698346.643161:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698346.643163:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698346.643167:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6248 00000100:00080000:3.0:1787698346.643170:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698346.643173:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698346.643178:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698346.643191:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698346.643191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134579072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698346.643202:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698346.643206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698346.643207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698346.643211:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6248 00000100:00080000:3.0:1787698346.643215:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698346.643218:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698346.643222:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698346.643232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134579200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698346.643237:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698346/real 1787698346] req@ffff9d85c0fb1c00 x1874531134578816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698346.643246:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698346.643251:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698346.643253:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698346.643257:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134578816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698346.643260:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6248 00000100:00080000:1.0:1787698346.643270:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698346.643274:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698346.643275:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698346.643281:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134578816:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698346.643286:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698346.643292:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698346.643303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134579328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698346.643308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578944:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698346.643319:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698346.643323:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698346.643325:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698346.643330:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6248 00000100:00080000:3.0:1787698346.643334:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698346.643339:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698346.643340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787698346.643347:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698346.643357:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134579456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698346.643365:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698346.643369:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698346/real 1787698346] req@ffff9d85c0fb0a80 x1874531134578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698346.643370:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698346.643373:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698346.643380:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6248 00000100:00080000:3.0:1787698346.643386:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698346.643390:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698346.643395:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698346.643412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698346.643417:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134579584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698346.643428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698346.643503:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698346.643506:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134578944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698346.643511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698346.643533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698346.643545:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698346/real 1787698346] req@ffff9d85c0fb3b80 x1874531134579072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698346.643564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134579072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698346.643572:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698346.643575:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698346.643578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698346.643584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698346.643604:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698346.643612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698346/real 1787698346] req@ffff9d85c0fb0380 x1874531134579200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698346.643619:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134579200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698346.643624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698346.643626:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698346.643627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698346.643637:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134579328:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698346.643648:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698346.643654:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698346/real 1787698346] req@ffff9d85c0fb2d80 x1874531134579328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698346.643665:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134579328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698346.643672:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698346.643675:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698346.643678:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134579328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698346.643684:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134579456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698346.643701:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698346.643711:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698346/real 1787698346] req@ffff9d85c0fb3800 x1874531134579456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698346.643721:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134579456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698346.643727:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698346.643730:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698346.643733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134579456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698346.643737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134579584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698346.643752:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698346.643761:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698346/real 1787698346] req@ffff9d85c0fb0000 x1874531134579584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698346.643771:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134579584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698346.643782:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698346.643784:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698346.643787:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134579584:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698347.219020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698348.243295:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698349.267054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698350.290975:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698351.315014:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698351.762963:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698351.762984:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698351.762987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698351.762997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6253 00000100:00080000:3.0:1787698351.763006:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698351.763013:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698351.763047:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134579712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698351.763076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698351.763079:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698351.763081:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698351.763085:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6253 00000100:00080000:3.0:1787698351.763088:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698351.763091:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698351.763099:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698351.763112:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134579840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698351.763120:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134579712:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698351.763125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698351.763129:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698351.763131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698351.763135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6253 00000100:00080000:3.0:1787698351.763139:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698351.763142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698351.763147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698351.763159:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134579968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698351.763167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698351.763170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698351.763173:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698351.763174:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698351.763180:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6253 00000100:00080000:3.0:1787698351.763184:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698351.763188:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698351.763193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698351.763206:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134580096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698351.763208:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698351/real 1787698351] req@ffff9d85c0fb2a00 x1874531134579712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698351.763234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698351.763237:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134579712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698351.763240:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698351.763244:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698351.763250:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698351.763251:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6253 00000100:00080000:1.0:1787698351.763255:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698351.763258:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698351.763259:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134579712:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698351.763267:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698351.763273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698351.763281:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698351.763287:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698351.763297:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698351.763298:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698351.763302:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698351.763305:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698351.763308:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698351/real 1787698351] req@ffff9d85c0fb2680 x1874531134579840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698351.763312:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6253 00000100:00080000:3.0:1787698351.763316:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698351.763322:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698351.763330:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134579840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698351.763330:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698351.763341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698351.763344:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698351.763350:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698351.763351:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134580352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698351.763357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579968:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698351.763369:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698351.763372:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698351.763376:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698351.763377:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698351.763383:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6253 00000100:00080000:3.0:1787698351.763386:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698351.763391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698351/real 1787698351] req@ffff9d85c0fb0e00 x1874531134579968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698351.763392:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698351.763401:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698351.763405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134579968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698351.763411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698351.763413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698351.763415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134579968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698351.763419:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698351.763424:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134580480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698351.763437:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698351.763450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698351/real 1787698351] req@ffff9d85c0fb0000 x1874531134580096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698351.763462:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134580096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698351.763480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698351.763482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698351.763484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698351.763495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134580224:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698351.763509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698351.763515:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698351/real 1787698351] req@ffff9d85c0fb3800 x1874531134580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698351.763522:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698351.763527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698351.763529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698351.763530:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134580224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698351.763533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134580352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698351.763541:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698351.763546:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698351/real 1787698351] req@ffff9d85c0fb2d80 x1874531134580352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698351.763559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134580352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698351.763563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698351.763565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698351.763567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134580352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698351.763569:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134580480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698351.763579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698351.763584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698351/real 1787698351] req@ffff9d85c0fb0380 x1874531134580480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698351.763590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134580480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698351.763599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698351.763600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698351.763602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134580480:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698352.339029:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698353.362984:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698354.387019:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698355.411053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698356.435039:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698356.882985:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698356.882992:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698356.883000:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698356.883008:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6258 00000100:00080000:1.0:1787698356.883016:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698356.883022:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698356.883050:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134580608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698356.883070:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698356.883074:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698356.883075:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698356.883080:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6258 00000100:00080000:1.0:1787698356.883083:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698356.883087:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698356.883096:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698356.883102:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134580608:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698356.883111:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134580736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698356.883128:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698356.883131:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698356.883133:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698356.883137:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6258 00000100:00080000:1.0:1787698356.883141:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698356.883145:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698356.883150:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698356.883162:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134580864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698356.883170:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698356.883174:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698356.883175:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698356.883179:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6258 00000100:00080000:1.0:1787698356.883182:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698356.883185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698356.883189:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698356.883196:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698356.883208:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134580992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698356.883216:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698356.883219:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698356.883221:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698356.883228:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6258 00000100:00100000:0.0:1787698356.883229:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698356/real 1787698356] req@ffff9d85b9d2f800 x1874531134580608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698356.883232:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698356.883240:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698356.883249:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698356.883256:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134580608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698356.883263:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134581120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698356.883274:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698356.883274:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698356.883280:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698356.883282:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698356.883283:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698356.883287:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6258 00000100:00080000:1.0:1787698356.883290:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698356.883292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134580608:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698356.883295:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698356.883310:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698356.883318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698356.883324:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134581248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698356.883335:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698356.883339:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698356.883341:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698356.883346:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698356.883349:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6258 00000100:00080000:1.0:1787698356.883360:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698356.883363:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698356/real 1787698356] req@ffff9d85b9d2fb80 x1874531134580736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698356.883367:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698356.883375:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698356.883384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134580736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698356.883390:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134581376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698356.883400:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698356.883407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698356.883413:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698356.883420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698356.883441:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698356.883452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698356/real 1787698356] req@ffff9d85b9d2e300 x1874531134580864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698356.883463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134580864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698356.883471:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698356.883474:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698356.883476:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698356.883480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698356.883493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698356.883499:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698356/real 1787698356] req@ffff9d85b9d2ce00 x1874531134580992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698356.883506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134580992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698356.883511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698356.883513:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698356.883514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134580992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698356.883517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134581120:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698356.883525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698356.883530:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698356/real 1787698356] req@ffff9d85b9d2c000 x1874531134581120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698356.883540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134581120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698356.883545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698356.883546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698356.883548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134581120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698356.883559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134581248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698356.883568:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698356.883573:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698356/real 1787698356] req@ffff9d85b9d2ca80 x1874531134581248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698356.883579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134581248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698356.883585:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698356.883588:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698356.883591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134581248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698356.883595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134581376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698356.883612:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698356.883621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698356/real 1787698356] req@ffff9d85b9d2f480 x1874531134581376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698356.883632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134581376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698356.883644:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698356.883647:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698356.883650:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134581376:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698357.459039:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698358.483019:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698359.507004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698360.530976:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698361.555009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698362.002973:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698362.002982:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698362.002986:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698362.002995:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6263 00000100:00080000:1.0:1787698362.003004:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698362.003010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698362.003052:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134581504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698362.003136:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134581504:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787698362.003194:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698362.003231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698362.003234:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698362.003236:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698362.003240:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6263 00000100:00080000:1.0:1787698362.003244:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698362.003247:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698362.003255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698362.003267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134581632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698362.003446:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698362/real 1787698362] req@ffff9d85b9d2f480 x1874531134581504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698362.003463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134581504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698362.003472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698362.003476:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698362.003479:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134581504:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698362.003501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134581632:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698362.003522:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698362.003532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698362/real 1787698362] req@ffff9d85b9d2ca80 x1874531134581632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698362.003543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134581632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698362.003550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698362.003553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698362.003556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134581632:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698362.003609:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698362.003613:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698362.003619:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698362.003622:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6263 00000100:00080000:1.0:1787698362.003626:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698362.003630:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698362.003636:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698362.003648:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134581760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698362.003667:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698362.003670:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698362.003671:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698362.003674:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6263 00000100:00080000:1.0:1787698362.003677:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698362.003680:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698362.003684:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698362.003686:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134581760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698362.003693:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134581888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698362.003700:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698362.003703:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698362.003704:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698362.003707:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6263 00000100:00080000:1.0:1787698362.003712:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698362.003716:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698362.003719:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698362.003725:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698362.003730:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698362/real 1787698362] req@ffff9d85b9d2c000 x1874531134581760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698362.003739:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134582016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698362.003743:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134581760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698362.003764:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698362.003765:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698362.003770:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698362.003773:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698362.003773:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698362.003779:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6263 00000100:00100000:0.0:1787698362.003782:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134581760:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787698362.003784:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698362.003791:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698362.003814:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698362.003821:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134581888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698362.003825:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134582144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698362.003833:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698362.003835:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698362.003837:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698362.003839:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6263 00000100:00080000:1.0:1787698362.003847:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698362.003867:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698362.003870:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698362.003877:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134582272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698362.003889:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698362.003912:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698362/real 1787698362] req@ffff9d85b9d2ce00 x1874531134581888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698362.003927:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134581888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698362.003933:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698362.003935:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698362.003937:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134581888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698362.003941:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698362.003955:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698362.003962:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698362/real 1787698362] req@ffff9d85b9d2e300 x1874531134582016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698362.003969:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134582016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698362.003974:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698362.003975:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698362.003977:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698362.003987:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698362.003996:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698362.004001:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698362/real 1787698362] req@ffff9d85b9d2fb80 x1874531134582144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698362.004008:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134582144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698362.004012:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698362.004013:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698362.004015:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698362.004018:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698362.004026:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698362.004030:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698362/real 1787698362] req@ffff9d85b9d2f800 x1874531134582272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698362.004036:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134582272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698362.004044:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698362.004046:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698362.004048:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582272:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698362.579019:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698363.603023:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698364.627021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698365.650986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698366.675019:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698367.122950:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698367.122958:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698367.122962:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698367.122971:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6268 00000100:00080000:3.0:1787698367.122980:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698367.122987:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698367.123022:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134582400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698367.123053:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698367.123057:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698367.123059:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698367.123063:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6268 00000100:00080000:3.0:1787698367.123067:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698367.123071:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698367.123080:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698367.123091:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134582400:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698367.123094:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134582528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698367.123107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698367.123110:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698367.123112:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698367.123115:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6268 00000100:00080000:3.0:1787698367.123119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698367.123123:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698367.123128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698367.123139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134582656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698367.123155:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698367.123158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698367.123162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698367.123164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698367.123169:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6268 00000100:00080000:3.0:1787698367.123174:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698367.123177:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698367.123183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698367.123192:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698367/real 1787698367] req@ffff9d85c0fb3b80 x1874531134582400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698367.123197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134582784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698367.123211:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698367.123212:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134582400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698367.123217:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698367.123219:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698367.123224:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6268 00000100:00080000:3.0:1787698367.123230:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698367.123235:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698367.123237:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698367.123240:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698367.123244:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134582400:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698367.123245:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698367.123258:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134582912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698367.123266:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134582528:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698367.123270:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698367.123275:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698367.123277:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698367.123282:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6268 00000800:00000100:0.0:1787698367.123286:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698367.123287:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698367.123294:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698367.123297:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698367/real 1787698367] req@ffff9d85c0fb0a80 x1874531134582528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698367.123300:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698367.123310:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134582528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698367.123317:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134583040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698367.123325:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698367.123332:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698367.123338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134582528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698367.123349:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134582656:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698367.123349:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698367.123356:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698367.123359:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698367.123364:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6268 00000800:00000100:0.0:1787698367.123368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698367.123368:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698367.123375:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698367.123377:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698367/real 1787698367] req@ffff9d85c0fb1c00 x1874531134582656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698367.123381:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698367.123386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134582656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698367.123393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698367.123397:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134583168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698367.123398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698367.123403:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134582656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698367.123414:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134582784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698367.123427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698367.123433:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698367/real 1787698367] req@ffff9d85c0fb1f80 x1874531134582784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698367.123439:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134582784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698367.123444:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698367.123446:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698367.123447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134582784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698367.123456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582912:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698367.123465:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698367.123470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698367/real 1787698367] req@ffff9d85c0fb3100 x1874531134582912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698367.123477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134582912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698367.123481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698367.123483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698367.123484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134582912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698367.123488:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698367.123496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698367.123500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698367/real 1787698367] req@ffff9d85c0fb1880 x1874531134583040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698367.123507:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134583040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698367.123511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698367.123513:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698367.123514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698367.123517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698367.123524:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698367.123528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698367/real 1787698367] req@ffff9d85c0fb1500 x1874531134583168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698367.123534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134583168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698367.123541:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698367.123543:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698367.123545:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583168:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698367.698978:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698368.723027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698369.747046:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698370.771005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698371.795002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698372.242958:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698372.242966:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698372.242968:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698372.242976:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6274 00000100:00080000:1.0:1787698372.242985:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698372.242990:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698372.243019:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134583296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698372.243039:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698372.243045:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698372.243047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698372.243051:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6274 00000100:00080000:1.0:1787698372.243054:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698372.243057:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698372.243065:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698372.243078:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134583424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698372.243081:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134583296:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698372.243086:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698372.243089:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698372.243091:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698372.243094:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6274 00000100:00080000:1.0:1787698372.243096:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698372.243099:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698372.243103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698372.243110:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134583552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698372.243117:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698372.243119:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698372.243121:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698372.243123:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6274 00000100:00080000:1.0:1787698372.243125:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698372.243128:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698372.243130:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787698372.243132:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698372.243139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134583680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698372.243147:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698372.243149:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698372.243150:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698372.243153:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6274 00000100:00080000:1.0:1787698372.243155:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698372.243158:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698372.243162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698372.243165:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698372/real 1787698372] req@ffff9d85b9d2dc00 x1874531134583296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698372.243169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134583808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698372.243177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698372.243179:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134583296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698372.243180:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698372.243181:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698372.243185:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6274 00000100:00080000:1.0:1787698372.243187:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698372.243189:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698372.243190:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698372.243192:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:1.0:1787698372.243193:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698372.243196:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134583296:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698372.243200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134583936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698372.243207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698372.243210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698372.243212:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698372.243215:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134583424:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698372.243215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6274 00000100:00080000:1.0:1787698372.243217:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698372.243220:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698372.243223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698372.243232:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698372.243232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134584064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698372.243241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698372/real 1787698372] req@ffff9d85b9d2e680 x1874531134583424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698372.243251:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134583424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698372.243259:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698372.243261:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698372.243265:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134583424:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698372.243271:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134583552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698372.243287:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698372.243295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698372/real 1787698372] req@ffff9d85b9d2d500 x1874531134583552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698372.243301:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134583552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698372.243306:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698372.243308:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698372.243310:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134583552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698372.243313:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134583680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698372.243322:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698372.243326:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698372/real 1787698372] req@ffff9d85b9d2ed80 x1874531134583680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698372.243332:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134583680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698372.243336:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698372.243337:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698372.243339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134583680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698372.243343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583808:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698372.243357:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698372.243367:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698372/real 1787698372] req@ffff9d85b9d2c700 x1874531134583808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698372.243378:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134583808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698372.243386:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698372.243389:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698372.243393:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698372.243398:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698372.243415:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698372.243424:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698372/real 1787698372] req@ffff9d85b9d2c380 x1874531134583936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698372.243435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134583936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698372.243443:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698372.243446:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698372.243449:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134583936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698372.243465:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698372.243480:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698372.243490:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698372/real 1787698372] req@ffff9d85b9d2dc00 x1874531134584064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698372.243501:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134584064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698372.243515:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698372.243518:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698372.243521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584064:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698372.818975:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698373.843061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698374.867010:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698375.891054:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698376.915026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698377.362925:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698377.362933:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698377.362936:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698377.362944:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6279 00000100:00080000:1.0:1787698377.362953:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698377.362958:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698377.362989:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134584192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698377.363015:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698377.363018:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698377.363019:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698377.363023:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6279 00000100:00080000:1.0:1787698377.363025:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698377.363027:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698377.363034:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698377.363042:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134584320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698377.363054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698377.363056:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698377.363058:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698377.363060:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6279 00000100:00080000:1.0:1787698377.363062:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698377.363064:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698377.363068:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698377.363079:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134584192:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698377.363079:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134584448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698377.363098:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698377.363100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698377.363101:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698377.363104:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6279 00000100:00080000:1.0:1787698377.363106:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698377.363109:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698377.363111:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698377.363118:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134584576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698377.363124:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698377.363126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698377.363127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698377.363130:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6279 00000100:00080000:1.0:1787698377.363133:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698377.363136:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698377.363139:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698377.363146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134584704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698377.363151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698377.363154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698377.363155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698377.363159:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6279 00000100:00080000:1.0:1787698377.363161:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698377.363163:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698377.363175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698377.363192:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134584320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698377.363196:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134584832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698377.363209:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698377.363212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698377.363215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698377.363219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6279 00000100:00080000:1.0:1787698377.363223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698377.363226:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698377.363231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698377.363244:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134584960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698377.363248:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134584448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698377.363280:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134584576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698377.363297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698377.363310:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698377.363337:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584960:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698377.939032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698378.963041:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698379.986991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698380.435128:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698380.435183:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698380.435200:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698380.435239:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698380.435249:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698380.435256:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698380.435266:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698380.435283:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698380.435295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698377/real 1787698380] req@ffff9d85b9d2e300 x1874531134584448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698380.435296:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698380.435324:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134584448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698380.435335:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698380.435340:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698380.435344:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134584448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698380.435352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698377/real 1787698380] req@ffff9d85b9d2ce00 x1874531134584576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698380.435367:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134584576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698380.435377:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698380.435380:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698380.435384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134584576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698380.435391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698377/real 1787698380] req@ffff9d85b9d2c000 x1874531134584704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698380.435403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134584704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698380.435410:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698380.435414:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698380.435417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698380.435423:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698377/real 1787698380] req@ffff9d85b9d2ca80 x1874531134584832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698380.435435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134584832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698380.435451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698380.435455:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698380.435459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698380.435466:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698377/real 1787698380] req@ffff9d85b9d2f800 x1874531134584192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698380.435517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134584192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698380.435525:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698380.435528:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698380.435532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134584192:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698380.435538:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698377/real 1787698380] req@ffff9d85b9d2fb80 x1874531134584320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698380.435551:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134584320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698380.435558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698380.435562:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698380.435565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134584320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698380.435595:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698377/real 1787698380] req@ffff9d85b9d2f480 x1874531134584960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698380.435621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134584960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698380.435630:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698380.435634:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698380.435637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134584960:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698381.011016:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698382.034977:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698382.482977:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698382.482986:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698382.482989:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698382.482998:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6284 00000100:00080000:3.0:1787698382.483006:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698382.483012:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698382.483045:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134585088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698382.483076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698382.483080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698382.483082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698382.483088:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6284 00000100:00080000:3.0:1787698382.483092:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698382.483096:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698382.483107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698382.483122:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134585216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698382.483129:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134585088:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698382.483137:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698382.483141:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698382.483143:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698382.483148:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6284 00000100:00080000:3.0:1787698382.483151:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698382.483156:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698382.483162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698382.483174:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134585344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698382.483184:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698382.483188:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698382.483190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698382.483194:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6284 00000100:00080000:3.0:1787698382.483198:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698382.483202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698382.483207:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698382.483210:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698382.483226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134585472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698382.483238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698382.483242:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698382.483244:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698382.483248:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6284 00000100:00080000:3.0:1787698382.483252:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698382.483259:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698382/real 1787698382] req@ffff9d85c0fb0700 x1874531134585088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698382.483260:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698382.483269:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698382.483287:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134585600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698382.483292:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134585088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698382.483313:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698382.483318:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698382.483319:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698382.483324:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698382.483329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698382.483334:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6284 00000100:00100000:0.0:1787698382.483339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134585088:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698382.483341:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698382.483351:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698382.483357:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698382.483369:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134585728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698382.483375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134585216:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698382.483389:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698382.483394:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698382.483396:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698382.483402:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6284 00000100:00080000:3.0:1787698382.483408:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698382.483413:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698382.483416:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698382.483424:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698382.483435:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698382/real 1787698382] req@ffff9d85c0fb3480 x1874531134585216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698382.483440:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134585856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698382.483483:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134585216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698382.483493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698382.483497:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698382.483500:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134585216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698382.483508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134585344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698382.483530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698382.483540:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698382/real 1787698382] req@ffff9d85c0fb2300 x1874531134585344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698382.483557:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134585344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698382.483565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698382.483574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698382.483577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134585344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698382.483583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134585472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698382.483600:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698382.483609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698382/real 1787698382] req@ffff9d85c0fb1500 x1874531134585472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698382.483620:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134585472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698382.483629:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698382.483632:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698382.483636:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134585472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698382.483642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134585600:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698382.483663:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698382.483674:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698382/real 1787698382] req@ffff9d85c0fb1880 x1874531134585600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698382.483687:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134585600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698382.483703:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698382.483707:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698382.483710:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134585600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698382.483729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134585728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698382.483751:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698382.483762:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698382/real 1787698382] req@ffff9d85c0fb3100 x1874531134585728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698382.483773:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134585728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698382.483781:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698382.483784:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698382.483787:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134585728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698382.483793:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134585856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698382.483811:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698382.483821:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698382/real 1787698382] req@ffff9d85c0fb1f80 x1874531134585856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698382.483831:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134585856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698382.483840:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698382.483843:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698382.483847:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134585856:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698383.059016:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698384.083093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698385.107011:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698386.131087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698387.155053:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698387.602951:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698387.602959:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698387.602963:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698387.602970:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6289 00000100:00080000:3.0:1787698387.602980:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698387.602985:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698387.603013:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134585984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698387.603035:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698387.603039:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698387.603040:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698387.603044:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6289 00000100:00080000:3.0:1787698387.603047:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698387.603051:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698387.603059:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698387.603071:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134586112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698387.603080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698387.603083:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698387.603085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698387.603086:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134585984:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698387.603088:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6289 00000100:00080000:3.0:1787698387.603091:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698387.603094:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698387.603098:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698387.603106:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134586240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698387.603113:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698387.603116:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698387.603117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698387.603120:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6289 00000100:00080000:3.0:1787698387.603122:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698387.603125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698387.603128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698387.603137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134586368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698387.603138:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698387.603144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698387.603146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698387.603147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698387.603150:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6289 00000100:00080000:3.0:1787698387.603152:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698387.603156:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698387.603160:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698387.603167:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134586496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698387.603174:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698387/real 1787698387] req@ffff9d85c0fb1c00 x1874531134585984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698387.603175:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698387.603178:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698387.603180:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698387.603183:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6289 00000100:00080000:3.0:1787698387.603185:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698387.603189:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134585984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698387.603189:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698387.603192:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698387.603198:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698387.603200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134586624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698387.603202:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698387.603206:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134585984:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698387.603207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698387.603210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698387.603211:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698387.603214:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6289 00000100:00080000:3.0:1787698387.603217:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698387.603219:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698387.603223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698387.603231:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134586752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698387.603237:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134586112:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698387.603261:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698387.603274:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698387/real 1787698387] req@ffff9d85c0fb0a80 x1874531134586112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698387.603287:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134586112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698387.603296:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698387.603299:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698387.603302:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134586112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698387.603308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134586240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698387.603337:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698387.603347:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698387/real 1787698387] req@ffff9d85c0fb3b80 x1874531134586240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698387.603376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134586240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698387.603382:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698387.603384:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698387.603386:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134586240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698387.603389:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134586368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698387.603403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698387.603409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698387/real 1787698387] req@ffff9d85c0fb0380 x1874531134586368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698387.603422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134586368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698387.603427:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698387.603429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698387.603431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134586368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698387.603434:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134586496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698387.603444:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698387.603449:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698387/real 1787698387] req@ffff9d85c0fb2d80 x1874531134586496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698387.603456:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134586496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698387.603460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698387.603461:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698387.603463:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134586496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698387.603466:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134586624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698387.603474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698387.603478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698387/real 1787698387] req@ffff9d85c0fb3800 x1874531134586624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698387.603484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134586624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698387.603488:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698387.603489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698387.603491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134586624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698387.603501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134586752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698387.603509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698387.603514:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698387/real 1787698387] req@ffff9d85c0fb0000 x1874531134586752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698387.603524:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134586752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698387.603528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698387.603530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698387.603532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134586752:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698388.179077:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698389.203032:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698390.226964:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698391.251003:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698392.275018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698392.722989:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698392.723001:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698392.723008:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698392.723021:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6294 00000100:00080000:3.0:1787698392.723034:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698392.723042:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698392.723093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134586880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698392.723126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698392.723130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698392.723132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698392.723137:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6294 00000100:00080000:3.0:1787698392.723140:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698392.723144:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698392.723156:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698392.723171:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134587008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698392.723182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698392.723186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698392.723188:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698392.723192:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6294 00000100:00080000:3.0:1787698392.723195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698392.723199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698392.723205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698392.723221:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134587136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698392.723223:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134586880:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698392.723247:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698392.723252:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698392.723253:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698392.723258:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6294 00000100:00080000:3.0:1787698392.723262:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698392.723283:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698392.723289:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698392.723307:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134587264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698392.723313:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698392.723328:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698392.723332:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698392.723334:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698392.723348:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6294 00000100:00080000:3.0:1787698392.723354:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698392.723360:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698392.723368:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698392/real 1787698392] req@ffff9d85c0fb0e00 x1874531134586880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698392.723369:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698392.723395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134586880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698392.723403:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134587392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698392.723413:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698392.723421:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698392.723423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698392.723435:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698392.723440:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698392.723441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134586880:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698392.723454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587008:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698392.723458:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6294 00000100:00080000:3.0:1787698392.723464:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698392.723470:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698392.723476:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698392.723483:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698392.723495:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134587520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698392.723503:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698392/real 1787698392] req@ffff9d85c0fb2680 x1874531134587008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698392.723511:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698392.723521:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698392.723528:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698392.723538:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6294 00000100:00100000:0.0:1787698392.723540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134587008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698392.723544:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698392.723556:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698392.723560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698392.723563:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698392.723568:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698392.723575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698392.723584:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134587648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698392.723605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698392.723634:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698392.723649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698392/real 1787698392] req@ffff9d85c0fb2a00 x1874531134587136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698392.723661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134587136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698392.723669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698392.723672:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698392.723675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698392.723682:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698392.723699:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698392.723710:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698392/real 1787698392] req@ffff9d85c0fb0000 x1874531134587264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698392.723721:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134587264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698392.723728:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698392.723732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698392.723735:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698392.723753:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134587392:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698392.723773:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698392.723789:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698392/real 1787698392] req@ffff9d85c0fb3800 x1874531134587392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698392.723799:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134587392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698392.723806:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698392.723809:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698392.723812:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134587392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698392.723818:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134587520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698392.723837:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698392.723847:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698392/real 1787698392] req@ffff9d85c0fb2d80 x1874531134587520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698392.723906:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134587520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698392.723914:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698392.723918:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698392.723921:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134587520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698392.723927:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134587648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698392.723945:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698392.723957:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698392/real 1787698392] req@ffff9d85c0fb0380 x1874531134587648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698392.723967:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134587648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698392.723975:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698392.723978:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698392.723982:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134587648:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698393.299027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698394.323046:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698395.347049:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698396.371035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698397.395036:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698397.843098:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698397.843109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698397.843113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698397.843123:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6299 00000100:00080000:0.0:1787698397.843134:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698397.843141:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698397.843178:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134587776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698397.843223:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698397.843226:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698397.843229:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698397.843237:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6299 00000100:00080000:0.0:1787698397.843241:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698397.843245:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698397.843254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698397.843267:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134587776:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698397.843270:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698397.843283:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698397.843287:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698397.843288:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698397.843292:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6299 00000100:00080000:0.0:1787698397.843296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698397.843301:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698397.843307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698397.843318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698397.843321:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134588032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698397.843332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698397.843335:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698397.843337:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698397.843341:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6299 00000100:00080000:0.0:1787698397.843345:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698397.843351:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698397.843351:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698397/real 1787698397] req@ffff9d85b9d08e00 x1874531134587776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698397.843359:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698397.843376:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134588160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698397.843377:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134587776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698397.843394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698397.843398:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698397.843400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698397.843405:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698397.843409:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698397.843409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134587776:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698397.843419:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6299 00000100:00080000:0.0:1787698397.843423:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698397.843427:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698397.843432:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698397.843434:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698397.843447:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134588288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698397.843456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698397.843459:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698397.843464:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698397.843466:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698397.843468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698397/real 1787698397] req@ffff9d85b9d08a80 x1874531134587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698397.843472:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6299 00000100:00080000:0.0:1787698397.843478:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698397.843484:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698397.843489:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698397.843489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698397.843502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698397.843503:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134588416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698397.843510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698397.843519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134587904:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698397.843521:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698397.843526:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698397.843529:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588032:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698397.843530:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698397.843538:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6299 00000100:00080000:0.0:1787698397.843541:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698397.843545:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698397.843551:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698397.843551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698397.843562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698397/real 1787698397] req@ffff9d85b9d08000 x1874531134588032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698397.843567:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134588544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698397.843573:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134588032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698397.843587:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698397.843590:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698397.843593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698397.843597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698397.843608:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698397.843618:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698397/real 1787698397] req@ffff9d85b9d0ad80 x1874531134588160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698397.843629:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134588160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698397.843637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698397.843639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698397.843642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698397.843656:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134588288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698397.843671:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698397.843678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698397/real 1787698397] req@ffff9d85b9d0bb80 x1874531134588288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698397.843684:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134588288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698397.843689:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698397.843691:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698397.843693:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134588288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698397.843696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134588416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698397.843704:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698397.843709:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698397/real 1787698397] req@ffff9d85b9d0a680 x1874531134588416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698397.843715:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134588416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698397.843719:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698397.843720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698397.843722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134588416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698397.843725:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134588544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698397.843732:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698397.843736:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698397/real 1787698397] req@ffff9d85b9d0b480 x1874531134588544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698397.843742:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134588544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698397.843746:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698397.843748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698397.843750:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134588544:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698398.419017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698399.442959:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698400.467026:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698401.491026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698402.514996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698402.963007:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698402.963020:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698402.963025:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698402.963037:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6304 00000100:00080000:0.0:1787698402.963050:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698402.963058:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698402.963118:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134588672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698402.963149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698402.963153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698402.963155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698402.963160:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6304 00000100:00080000:0.0:1787698402.963163:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698402.963168:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698402.963178:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698402.963193:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134588800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698402.963203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698402.963206:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698402.963208:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698402.963212:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6304 00000100:00080000:0.0:1787698402.963216:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698402.963225:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698402.963228:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134588672:192.168.201.137@tcp:250:kworker.0 00000008:00100000:0.0:1787698402.963234:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698402.963253:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134588928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698402.963266:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698402.963269:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698402.963271:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698402.963277:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6304 00000100:00080000:0.0:1787698402.963290:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698402.963298:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698402.963302:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698402.963304:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698402.963322:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134589056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698402.963333:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698402.963337:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698402.963339:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698402.963343:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6304 00000100:00100000:1.0:1787698402.963348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698402/real 1787698402] req@ffff9d85b9d0b800 x1874531134588672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698402.963350:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698402.963361:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698402.963373:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698402.963384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134588672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698402.963394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698402.963396:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698402.963401:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698402.963411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134588672:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698402.963429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588800:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698402.963435:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698402.963441:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698402.963443:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698402.963449:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6304 00000800:00000100:1.0:1787698402.963454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698402.963457:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698402.963465:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698402.963470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698402/real 1787698402] req@ffff9d85b9d08700 x1874531134588800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698402.963474:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698402.963491:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134589312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698402.963499:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134588800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698402.963510:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698402.963520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698402.963527:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698402.963529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698402.963532:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698402.963535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588800:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698402.963544:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6304 00000100:00080000:0.0:1787698402.963551:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698402.963562:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698402.963569:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698402.963573:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698402.963584:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134589440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698402.963604:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698402.963622:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698402/real 1787698402] req@ffff9d85b9d09500 x1874531134588928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698402.963643:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134588928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698402.963652:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698402.963655:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698402.963659:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134588928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698402.963664:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698402.963679:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698402.963687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698402/real 1787698402] req@ffff9d85b9d09180 x1874531134589056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698402.963694:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134589056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698402.963699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698402.963701:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698402.963703:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698402.963706:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134589184:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698402.963725:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698402.963731:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698402/real 1787698402] req@ffff9d85b9d0a300 x1874531134589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698402.963737:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698402.963742:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698402.963744:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698402.963746:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134589184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698402.963749:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134589312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698402.963764:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698402.963770:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698402/real 1787698402] req@ffff9d85b9d09f80 x1874531134589312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698402.963776:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134589312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698402.963781:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698402.963783:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698402.963786:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134589312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698402.963802:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134589440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698402.963819:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698402.963830:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698402/real 1787698402] req@ffff9d85b9d09c00 x1874531134589440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698402.963840:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134589440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698402.963847:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698402.963921:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698402.963927:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134589440:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698403.539046:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698404.562983:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698405.587005:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698406.610991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698407.635024:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698408.082930:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698408.082939:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698408.082943:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698408.082952:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6309 00000100:00080000:0.0:1787698408.082961:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698408.082968:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698408.083032:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134589568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698408.083055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698408.083059:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698408.083060:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698408.083064:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6309 00000100:00080000:0.0:1787698408.083068:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698408.083072:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698408.083083:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698408.083088:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134589568:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698408.083098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134589696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698408.083106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698408.083108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698408.083110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698408.083112:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6309 00000100:00080000:0.0:1787698408.083115:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698408.083117:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698408.083121:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698408.083127:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134589824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698408.083133:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698408.083135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698408.083137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698408.083140:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698408.083143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6309 00000100:00080000:0.0:1787698408.083146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698408.083149:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698408.083152:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698408.083160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134589952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698408.083165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698408.083167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698408.083168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698408.083172:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6309 00000100:00080000:0.0:1787698408.083174:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698408.083178:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698408/real 1787698408] req@ffff9d85b9d09c00 x1874531134589568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698408.083179:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698408.083186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698408.083195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134590080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698408.083199:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134589568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698408.083207:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698408.083213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698408.083215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698408.083215:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698408.083219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6309 00000100:00080000:1.0:1787698408.083221:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698408.083223:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698408.083226:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134589568:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698408.083228:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698408.083234:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698408.083241:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134590208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698408.083248:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698408.083250:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589696:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698408.083252:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698408.083255:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698408.083259:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6309 00000100:00080000:0.0:1787698408.083262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698408.083265:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698408.083268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698408.083272:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698408.083280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134590336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698408.083284:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698408/real 1787698408] req@ffff9d85b9d09f80 x1874531134589696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698408.083297:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134589696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698408.083302:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698408.083305:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698408.083307:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698408.083311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698408.083326:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698408.083337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698408/real 1787698408] req@ffff9d85b9d0a300 x1874531134589824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698408.083357:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134589824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698408.083365:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698408.083368:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698408.083371:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698408.083376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698408.083390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698408.083397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698408/real 1787698408] req@ffff9d85b9d09180 x1874531134589952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698408.083403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134589952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698408.083408:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698408.083410:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698408.083412:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134589952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698408.083415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590080:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698408.083423:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698408.083428:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698408/real 1787698408] req@ffff9d85b9d09500 x1874531134590080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698408.083435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134590080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698408.083451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698408.083455:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698408.083458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698408.083474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698408.083491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698408.083498:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698408/real 1787698408] req@ffff9d85b9d08700 x1874531134590208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698408.083504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134590208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698408.083510:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698408.083511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698408.083513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698408.083516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698408.083543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698408.083554:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698408/real 1787698408] req@ffff9d85b9d0b800 x1874531134590336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698408.083565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134590336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698408.083573:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698408.083575:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698408.083579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590336:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698408.659027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698409.683009:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698410.707012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698411.731024:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698412.755018:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698413.202945:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698413.202956:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698413.202960:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698413.202970:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6314 00000100:00080000:3.0:1787698413.202980:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698413.202987:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698413.203018:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134590464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698413.203038:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698413.203040:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698413.203041:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698413.203044:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6314 00000100:00080000:3.0:1787698413.203047:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698413.203050:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698413.203057:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698413.203066:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134590592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698413.203070:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134590464:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698413.203076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698413.203080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698413.203081:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698413.203084:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6314 00000100:00080000:3.0:1787698413.203086:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698413.203088:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698413.203091:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698413.203098:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134590720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698413.203103:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698413.203105:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698413.203106:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698413.203109:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6314 00000100:00080000:3.0:1787698413.203111:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698413.203114:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698413.203117:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698413.203124:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134590848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698413.203129:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698413.203131:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698413.203132:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698413.203137:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6314 00000800:00000100:1.0:1787698413.203139:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698413.203140:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698413.203144:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698413.203146:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698413.203152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134590976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698413.203158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698413.203160:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698413.203161:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698413.203164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6314 00000100:00080000:3.0:1787698413.203166:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698413.203169:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698413.203172:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698413.203177:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698413/real 1787698413] req@ffff9d85c0fb3b80 x1874531134590464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698413.203180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134591104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698413.203209:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698413.203213:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134590464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698413.203214:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698413.203217:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698413.203226:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6314 00000100:00080000:1.0:1787698413.203229:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698413.203231:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698413.203235:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698413.203239:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698413.203240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134590464:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698413.203248:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698413.203261:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134590592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698413.203262:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134591232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698413.203284:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698413.203295:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698413/real 1787698413] req@ffff9d85c0fb0a80 x1874531134590592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698413.203303:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134590592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698413.203311:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698413.203314:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698413.203317:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134590592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698413.203322:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134590720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698413.203339:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698413.203350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698413/real 1787698413] req@ffff9d85c0fb1c00 x1874531134590720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698413.203361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134590720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698413.203368:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698413.203369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698413.203371:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134590720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698413.203375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134590848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698413.203388:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698413.203394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698413/real 1787698413] req@ffff9d85c0fb1f80 x1874531134590848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698413.203400:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134590848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698413.203405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698413.203407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698413.203408:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134590848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698413.203411:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590976:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698413.203419:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698413.203424:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698413/real 1787698413] req@ffff9d85c0fb3100 x1874531134590976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698413.203430:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134590976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698413.203434:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698413.203436:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698413.203438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134590976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698413.203440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134591104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698413.203448:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698413.203453:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698413/real 1787698413] req@ffff9d85c0fb1880 x1874531134591104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698413.203458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134591104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698413.203462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698413.203463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698413.203465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134591104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698413.203475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134591232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698413.203484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698413.203489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698413/real 1787698413] req@ffff9d85c0fb1500 x1874531134591232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698413.203495:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134591232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698413.203499:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698413.203500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698413.203503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134591232:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698413.779032:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698414.802997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698415.827067:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698416.850956:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698417.875013:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698418.322962:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698418.322970:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698418.322974:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698418.322983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6320 00000100:00080000:1.0:1787698418.322994:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698418.323000:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698418.323037:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134591360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698418.323059:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698418.323061:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698418.323063:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698418.323065:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6320 00000100:00080000:1.0:1787698418.323068:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698418.323070:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698418.323077:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698418.323086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134591488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698418.323093:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698418.323095:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698418.323097:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698418.323101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134591360:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698418.323101:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6320 00000100:00080000:1.0:1787698418.323103:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698418.323108:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698418.323113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698418.323118:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134591616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698418.323123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698418.323125:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698418.323126:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698418.323128:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6320 00000100:00080000:1.0:1787698418.323130:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698418.323131:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698418.323134:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698418.323141:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134591744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698418.323146:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698418.323148:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698418.323149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698418.323151:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6320 00000100:00080000:1.0:1787698418.323159:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698418.323162:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698418.323164:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698418.323168:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698418.323172:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134591872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698418.323178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698418.323181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698418.323182:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698418.323184:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6320 00000100:00080000:1.0:1787698418.323187:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698418.323189:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698418.323191:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698418.323197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134592000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698418.323202:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698418.323206:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698418/real 1787698418] req@ffff9d85b9d2f480 x1874531134591360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.323206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698418.323208:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698418.323215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6320 00000100:00080000:1.0:1787698418.323219:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698418.323224:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698418.323227:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698418.323231:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134591360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.323235:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134592128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698418.323248:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698418.323253:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698418.323256:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134591360:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698418.323273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134591488:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698418.323286:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698418.323294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698418/real 1787698418] req@ffff9d85b9d2fb80 x1874531134591488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698418.323301:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134591488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698418.323305:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698418.323307:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698418.323309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134591488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698418.323312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134591616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698418.323323:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698418.323328:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698418/real 1787698418] req@ffff9d85b9d2f800 x1874531134591616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698418.323334:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134591616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698418.323339:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698418.323340:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698418.323342:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134591616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698418.323345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134591744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698418.323353:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698418.323357:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698418/real 1787698418] req@ffff9d85b9d2ca80 x1874531134591744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698418.323363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134591744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698418.323367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698418.323369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698418.323370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134591744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698418.323373:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134591872:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698418.323381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698418.323385:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698418/real 1787698418] req@ffff9d85b9d2c000 x1874531134591872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698418.323395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134591872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698418.323399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698418.323400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698418.323402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134591872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698418.323404:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698418.323412:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698418.323416:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698418/real 1787698418] req@ffff9d85b9d2ce00 x1874531134592000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698418.323422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134592000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698418.323426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698418.323427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698418.323429:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698418.323432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698418.323439:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698418.323444:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698418/real 1787698418] req@ffff9d85b9d2e300 x1874531134592128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698418.323449:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134592128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698418.323457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698418.323459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698418.323461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592128:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698418.898991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698419.923019:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698420.946986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698421.971055:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698422.995037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698423.442968:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698423.442978:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698423.442983:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698423.442993:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6325 00000100:00080000:1.0:1787698423.443005:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698423.443012:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698423.443050:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134592256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698423.443081:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698423.443085:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698423.443087:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698423.443092:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6325 00000100:00080000:1.0:1787698423.443095:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698423.443099:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698423.443109:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698423.443123:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134592384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698423.443133:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698423.443138:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698423.443140:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698423.443145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6325 00000100:00100000:0.0:1787698423.443150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134592256:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698423.443151:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698423.443160:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698423.443167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698423.443181:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134592512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698423.443192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698423.443196:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698423.443199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698423.443202:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6325 00000100:00080000:1.0:1787698423.443209:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698423.443213:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698423.443218:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698423.443227:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698423.443232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134592640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698423.443244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698423.443248:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698423.443250:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698423.443255:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6325 00000100:00080000:1.0:1787698423.443259:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698423.443264:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698423.443270:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698423/real 1787698423] req@ffff9d85b9d2dc00 x1874531134592256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698423.443273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698423.443291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134592768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698423.443295:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134592256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698423.443317:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698423.443322:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698423.443323:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698423.443328:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698423.443332:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698423.443337:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6325 00000100:00100000:0.0:1787698423.443340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134592256:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698423.443345:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698423.443350:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134592384:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698423.443355:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698423.443362:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698423.443376:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698423.443376:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134592896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698423.443388:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698423/real 1787698423] req@ffff9d85b9d2c380 x1874531134592384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698423.443417:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698423.443423:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134592384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698423.443424:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698423.443428:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698423.443437:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6325 00000100:00080000:0.0:1787698423.443443:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698423.443445:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698423.443453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698423.443454:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698423.443461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134592384:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787698423.443464:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698423.443483:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134593024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698423.443491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134592512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698423.443513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698423.443525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698423/real 1787698423] req@ffff9d85b9d2c700 x1874531134592512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698423.443534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134592512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698423.443540:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698423.443542:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698423.443544:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134592512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698423.443548:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134592640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698423.443558:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698423.443564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698423/real 1787698423] req@ffff9d85b9d2ed80 x1874531134592640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698423.443571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134592640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698423.443576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698423.443577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698423.443579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134592640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698423.443583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592768:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698423.443593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698423.443599:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698423/real 1787698423] req@ffff9d85b9d2d500 x1874531134592768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698423.443611:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134592768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698423.443619:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698423.443622:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698423.443626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698423.443632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698423.443652:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698423.443663:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698423/real 1787698423] req@ffff9d85b9d2e680 x1874531134592896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698423.443676:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134592896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698423.443684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698423.443687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698423.443691:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134592896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698423.443709:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698423.443729:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698423.443740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698423/real 1787698423] req@ffff9d85b9d2e300 x1874531134593024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698423.443781:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134593024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698423.443799:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698423.443803:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698423.443807:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593024:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698424.019044:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698425.043039:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698426.067014:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698427.091030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698428.114988:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698428.562989:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698428.562998:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698428.563002:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698428.563011:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6330 00000100:00080000:1.0:1787698428.563021:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698428.563028:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698428.563060:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134593152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698428.563083:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698428.563087:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698428.563089:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698428.563093:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6330 00000100:00080000:1.0:1787698428.563097:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698428.563101:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698428.563111:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698428.563128:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134593280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698428.563138:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698428.563141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698428.563142:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698428.563146:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6330 00000100:00080000:1.0:1787698428.563149:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698428.563153:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134593152:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698428.563154:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698428.563162:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698428.563173:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134593408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698428.563180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698428.563183:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698428.563184:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698428.563187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6330 00000100:00080000:1.0:1787698428.563190:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698428.563192:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698428.563195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698428.563202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134593536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698428.563210:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698428.563212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698428.563213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698428.563217:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6330 00000100:00080000:1.0:1787698428.563221:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698428.563222:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698428.563227:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698428.563231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698428.563239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134593664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698428.563246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698428.563248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698428.563250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698428.563252:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6330 00000100:00080000:1.0:1787698428.563254:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698428.563257:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698428.563261:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698428.563270:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698428/real 1787698428] req@ffff9d85b9d2ce00 x1874531134593152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698428.563270:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134593792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698428.563298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698428.563301:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698428.563303:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698428.563306:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6330 00000100:00080000:1.0:1787698428.563308:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698428.563312:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134593152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698428.563313:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698428.563320:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698428.563332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698428.563332:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134593920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698428.563342:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698428.563349:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134593152:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698428.563358:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134593280:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698428.563380:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698428.563392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698428/real 1787698428] req@ffff9d85b9d2c000 x1874531134593280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698428.563405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134593280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698428.563414:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698428.563417:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698428.563421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134593280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698428.563447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134593408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698428.563469:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698428.563480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698428/real 1787698428] req@ffff9d85b9d2ca80 x1874531134593408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698428.563492:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134593408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698428.563501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698428.563504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698428.563507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134593408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698428.563511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134593536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698428.563525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698428.563531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698428/real 1787698428] req@ffff9d85b9d2f800 x1874531134593536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698428.563539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134593536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698428.563544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698428.563545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698428.563547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134593536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698428.563551:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593664:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698428.563561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698428.563576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698428/real 1787698428] req@ffff9d85b9d2fb80 x1874531134593664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698428.563584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134593664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698428.563590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698428.563592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698428.563594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698428.563597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698428.563608:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698428.563614:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698428/real 1787698428] req@ffff9d85b9d2f480 x1874531134593792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698428.563621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134593792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698428.563627:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698428.563628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698428.563630:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698428.563633:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698428.563643:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698428.563648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698428/real 1787698428] req@ffff9d85b9d2e300 x1874531134593920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698428.563681:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134593920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698428.563692:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698428.563694:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698428.563696:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134593920:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698429.139029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698430.162986:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698431.187047:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698432.211038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698433.234997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698433.682977:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698433.682985:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698433.682988:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698433.682997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6335 00000100:00080000:3.0:1787698433.683007:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698433.683012:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698433.683045:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134594048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698433.683067:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698433.683070:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698433.683072:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698433.683076:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6335 00000100:00080000:3.0:1787698433.683081:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698433.683085:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698433.683094:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698433.683109:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134594176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698433.683121:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698433.683124:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698433.683127:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698433.683131:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6335 00000100:00080000:3.0:1787698433.683135:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698433.683139:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698433.683144:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698433.683159:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134594048:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698433.683160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134594304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698433.683184:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698433.683189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698433.683192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698433.683197:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6335 00000100:00080000:3.0:1787698433.683202:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698433.683206:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698433.683213:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698433.683229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134594432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698433.683244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698433.683246:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698433.683249:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698433.683251:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6335 00000100:00080000:3.0:1787698433.683256:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698433.683259:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698433.683262:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698433.683274:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698433.683274:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134594560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698433.683285:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698433.683288:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698433.683289:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698433.683292:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6335 00000100:00080000:3.0:1787698433.683294:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698433.683297:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698433.683300:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698433.683307:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134594688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698433.683314:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698433.683316:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698433.683318:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698433.683321:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6335 00000100:00080000:3.0:1787698433.683324:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698433.683327:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698433.683331:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698433.683343:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134594816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698433.683348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698433/real 1787698433] req@ffff9d85c0fb2300 x1874531134594048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698433.683388:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134594048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698433.683402:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698433.683407:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698433.683412:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134594048:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698433.683427:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134594176:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698433.683449:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698433.683459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698433/real 1787698433] req@ffff9d85c0fb3480 x1874531134594176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698433.683467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134594176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698433.683473:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698433.683475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698433.683477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134594176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698433.683501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134594304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698433.683517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698433.683527:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698433/real 1787698433] req@ffff9d85c0fb0700 x1874531134594304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698433.683539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134594304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698433.683547:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698433.683551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698433.683554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134594304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698433.683561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134594432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698433.683598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698433.683610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698433/real 1787698433] req@ffff9d85c0fb1500 x1874531134594432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698433.683621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134594432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698433.683630:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698433.683633:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698433.683637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134594432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698433.683643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134594560:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698433.683662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698433.683671:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698433/real 1787698433] req@ffff9d85c0fb1880 x1874531134594560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698433.683679:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134594560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698433.683692:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698433.683694:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698433.683696:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134594560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698433.683700:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134594688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698433.683714:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698433.683720:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698433/real 1787698433] req@ffff9d85c0fb3100 x1874531134594688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698433.683726:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134594688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698433.683731:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698433.683732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698433.683734:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134594688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698433.683737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134594816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698433.683747:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698433.683752:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698433/real 1787698433] req@ffff9d85c0fb1f80 x1874531134594816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698433.683758:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134594816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698433.683767:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698433.683769:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698433.683771:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134594816:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698434.259063:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698435.283017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698436.307006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698437.331057:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698438.354999:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698438.802990:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698438.803001:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698438.803006:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698438.803017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6340 00000100:00080000:0.0:1787698438.803027:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698438.803035:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698438.803081:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134594944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698438.803105:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698438.803108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698438.803109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698438.803112:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6340 00000100:00080000:0.0:1787698438.803115:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698438.803118:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698438.803127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698438.803136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134595072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698438.803144:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134594944:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698438.803145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698438.803153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698438.803156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698438.803160:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6340 00000100:00080000:0.0:1787698438.803162:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698438.803164:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698438.803168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698438.803179:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134595200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698438.803188:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698438.803203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698438.803210:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698438.803215:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6340 00000100:00080000:0.0:1787698438.803220:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698438.803227:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698438.803227:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698438.803236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698438.803250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134595328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698438.803258:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698438.803261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698438.803262:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698438.803266:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6340 00000100:00080000:0.0:1787698438.803269:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698438.803272:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698438.803275:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698438.803277:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698438/real 1787698438] req@ffff9d85b9d0b480 x1874531134594944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698438.803288:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134595456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698438.803298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698438.803302:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134594944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698438.803304:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698438.803308:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698438.803319:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6340 00000100:00080000:1.0:1787698438.803319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698438.803327:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698438.803329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698438.803337:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698438.803338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134594944:192.168.201.137@tcp:250:kworker.0 00000008:00100000:0.0:1787698438.803345:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698438.803360:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134595584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698438.803368:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595072:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698438.803369:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698438.803374:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698438.803376:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698438.803380:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6340 00000800:00000100:1.0:1787698438.803389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698438.803391:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698438.803397:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698438.803401:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698438.803403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698438/real 1787698438] req@ffff9d85b9d0a680 x1874531134595072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698438.803414:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134595712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698438.803417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134595072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698438.803438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698438.803442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698438.803444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698438.803449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698438.803464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698438.803471:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698438/real 1787698438] req@ffff9d85b9d0bb80 x1874531134595200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698438.803479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134595200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698438.803507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698438.803509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698438.803512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698438.803516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698438.803531:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698438.803539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698438/real 1787698438] req@ffff9d85b9d0ad80 x1874531134595328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698438.803546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134595328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698438.803551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698438.803554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698438.803556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698438.803559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134595456:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698438.803569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698438.803574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698438/real 1787698438] req@ffff9d85b9d08000 x1874531134595456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698438.803586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134595456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698438.803591:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698438.803593:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698438.803595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134595456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698438.803606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134595584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698438.803616:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698438.803622:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698438/real 1787698438] req@ffff9d85b9d08a80 x1874531134595584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698438.803629:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134595584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698438.803634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698438.803635:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698438.803637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134595584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698438.803641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134595712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698438.803650:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698438.803655:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698438/real 1787698438] req@ffff9d85b9d08e00 x1874531134595712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698438.803662:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134595712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698438.803671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698438.803673:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698438.803675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134595712:192.168.201.137@tcp:8:kworker.0 00010000:00010000:0.0:1787698439.379019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698440.402979:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698441.427021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698442.451047:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698443.474992:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698443.922981:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698443.922992:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698443.922997:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698443.923007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6345 00000100:00080000:2.0:1787698443.923018:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698443.923024:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698443.923064:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134595840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698443.923085:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698443.923089:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698443.923091:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698443.923095:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6345 00000100:00080000:2.0:1787698443.923099:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698443.923103:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698443.923114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698443.923125:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134595840:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698443.923135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134595968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698443.923152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698443.923155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698443.923157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698443.923160:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6345 00000100:00080000:2.0:1787698443.923163:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698443.923166:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698443.923171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698443.923180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134596096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698443.923189:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698443.923192:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698443.923194:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698443.923197:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6345 00000100:00080000:2.0:1787698443.923200:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698443.923204:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698443.923209:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698443.923221:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134596224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698443.923232:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698443.923235:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698443.923237:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698443.923241:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6345 00000100:00100000:1.0:1787698443.923243:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595968:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698443.923246:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698443.923251:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698443.923256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698443.923267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134596352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698443.923277:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596096:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698443.923277:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698443.923280:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698443.923282:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698443.923286:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6345 00000100:00080000:2.0:1787698443.923290:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698443.923294:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698443.923299:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698443.923300:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698443.923309:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134596480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698443.923319:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698443.923320:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134596352:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698443.923323:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698443.923325:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698443.923329:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6345 00000100:00080000:2.0:1787698443.923332:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698443.923336:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698443.923341:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698443.923343:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134596480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698443.923352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134596608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698443.923377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134596608:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698444.499025:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698445.523029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698446.547039:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698446.995249:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:3.0:1787698446.995317:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698446.995351:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698446.995407:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698446.995420:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698446.995429:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698446.995439:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698446.995448:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698446.995457:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698446.995578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698443/real 1787698446] req@ffff9d85babd5180 x1874531134596608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698446.995609:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134596608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698446.995624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698446.995630:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698446.995635:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134596608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698446.995644:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698443/real 1787698446] req@ffff9d85babd4e00 x1874531134596480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698446.995657:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134596480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698446.995664:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698446.995668:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698446.995671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134596480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698446.995679:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698443/real 1787698446] req@ffff9d85babd7b80 x1874531134596352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698446.995691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134596352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698446.995699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698446.995703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698446.995706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134596352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698446.995713:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698443/real 1787698446] req@ffff9d85babd4a80 x1874531134595968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698446.995726:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134595968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698446.995734:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698446.995739:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698446.995742:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134595968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698446.995749:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698443/real 1787698446] req@ffff9d85babd7100 x1874531134596096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698446.995762:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134596096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698446.995770:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698446.995774:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698446.995778:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698446.995785:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698443/real 1787698446] req@ffff9d85babd5f80 x1874531134596224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698446.995797:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134596224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698446.995804:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698446.995808:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698446.995811:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698446.995818:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698443/real 1787698446] req@ffff9d85babd4700 x1874531134595840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698446.995830:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134595840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698446.995838:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698446.995842:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698446.995846:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134595840:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787698447.570955:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698448.595097:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698449.043003:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698449.043016:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698449.043020:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698449.043031:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6350 00000100:00080000:2.0:1787698449.043042:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698449.043051:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698449.043085:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134596736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698449.043113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698449.043117:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698449.043119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698449.043123:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6350 00000100:00080000:2.0:1787698449.043127:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698449.043132:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698449.043144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698449.043160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698449.043183:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698449.043187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698449.043189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698449.043200:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6350 00000100:00100000:1.0:1787698449.043204:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134596736:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698449.043206:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698449.043213:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698449.043226:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698449.043246:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134596992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698449.043257:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698449.043261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698449.043265:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698449.043271:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6350 00000100:00080000:2.0:1787698449.043276:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698449.043280:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698449.043287:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698449.043300:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134597120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698449.043323:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698449.043327:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698449.043328:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698449.043334:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6350 00000100:00080000:2.0:1787698449.043340:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698449.043350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698449.043357:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698449.043365:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698449.043381:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134597248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698449.043394:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698449.043397:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698449.043399:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698449.043405:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6350 00000100:00080000:2.0:1787698449.043410:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698449.043416:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698449.043421:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698449.043433:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134597376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698449.043442:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698449.043446:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698449.043449:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698449.043452:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6350 00000100:00080000:2.0:1787698449.043455:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698449.043458:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698449.043463:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698449.043476:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134597504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698449.043562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698449/real 1787698449] req@ffff9d85babd7480 x1874531134596736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698449.043589:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134596736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698449.043602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698449.043607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698449.043616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134596736:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698449.043651:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596864:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698449.043681:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698449.043696:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698449/real 1787698449] req@ffff9d85babd5c00 x1874531134596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698449.043707:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698449.043715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698449.043718:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698449.043721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698449.043727:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698449.043752:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698449.043764:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698449/real 1787698449] req@ffff9d85babd4000 x1874531134596992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698449.043783:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134596992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698449.043791:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698449.043794:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698449.043798:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134596992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698449.043805:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134597120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698449.043826:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698449.043837:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698449/real 1787698449] req@ffff9d85babd4380 x1874531134597120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698449.043848:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134597120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698449.043904:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698449.043908:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698449.043911:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134597120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698449.043918:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134597248:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698449.043947:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698449.043960:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698449/real 1787698449] req@ffff9d85babd6300 x1874531134597248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698449.043970:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134597248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698449.043983:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698449.043987:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698449.043990:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134597248:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698449.043997:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134597376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698449.044016:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698449.044027:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698449/real 1787698449] req@ffff9d85babd6a00 x1874531134597376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698449.044038:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134597376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698449.044047:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698449.044050:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698449.044052:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134597376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698449.044057:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134597504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698449.044071:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698449.044079:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698449/real 1787698449] req@ffff9d85babd6680 x1874531134597504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698449.044093:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134597504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698449.044102:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698449.044105:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698449.044110:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134597504:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698449.619088:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698450.643103:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698451.667058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698452.691043:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698453.715030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698454.162955:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698454.162964:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698454.162968:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698454.162976:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6355 00000100:00080000:3.0:1787698454.162985:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698454.162991:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698454.163026:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134597632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698454.163049:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698454.163053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698454.163054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698454.163057:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6355 00000100:00080000:3.0:1787698454.163059:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698454.163061:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698454.163068:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698454.163076:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134597760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698454.163084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698454.163087:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698454.163088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698454.163091:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6355 00000100:00080000:3.0:1787698454.163093:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698454.163096:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698454.163099:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698454.163107:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134597888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698454.163109:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134597632:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698454.163118:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698454.163122:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698454.163123:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698454.163125:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6355 00000100:00080000:3.0:1787698454.163127:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698454.163128:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698454.163132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698454.163140:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134598016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698454.163147:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698454.163150:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698454.163151:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698454.163153:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6355 00000100:00080000:3.0:1787698454.163156:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698454.163158:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698454.163161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698454.163168:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698454.163174:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698454.163180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698454.163183:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698454.163184:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698454.163188:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6355 00000100:00080000:3.0:1787698454.163191:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698454.163194:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698454.163197:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698454.163204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134598272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698454.163209:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698454.163212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698454.163213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698454.163216:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6355 00000100:00080000:3.0:1787698454.163233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698454.163235:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698454/real 1787698454] req@ffff9d85c0fb1c00 x1874531134597632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698454.163239:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698454.163248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698454.163260:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134597632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698454.163261:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134598400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698454.163286:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698454.163291:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698454.163294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134597632:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698454.163308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134597760:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698454.163327:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698454.163335:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698454/real 1787698454] req@ffff9d85c0fb0a80 x1874531134597760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698454.163343:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134597760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698454.163349:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698454.163352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698454.163354:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134597760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698454.163371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134597888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698454.163382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698454.163388:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698454/real 1787698454] req@ffff9d85c0fb3b80 x1874531134597888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698454.163396:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134597888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698454.163401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698454.163402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698454.163404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134597888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698454.163408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698454.163417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698454.163423:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698454/real 1787698454] req@ffff9d85c0fb0380 x1874531134598016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698454.163430:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134598016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698454.163435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698454.163436:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698454.163438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698454.163441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134598144:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698454.163450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698454.163456:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698454/real 1787698454] req@ffff9d85c0fb2d80 x1874531134598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698454.163463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698454.163468:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698454.163470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698454.163472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134598144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698454.163475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134598272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698454.163484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698454.163489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698454/real 1787698454] req@ffff9d85c0fb3800 x1874531134598272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698454.163496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134598272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698454.163502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698454.163503:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698454.163505:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134598272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698454.163508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134598400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698454.163517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698454.163522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698454/real 1787698454] req@ffff9d85c0fb0000 x1874531134598400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698454.163528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134598400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698454.163533:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698454.163534:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698454.163537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134598400:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698454.739053:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698455.762963:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698456.787040:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698457.810996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698458.835030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698459.282964:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698459.282974:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698459.282979:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698459.282999:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6361 00000100:00080000:3.0:1787698459.283011:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698459.283020:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698459.283054:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134598528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698459.283084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698459.283086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698459.283088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698459.283091:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6361 00000100:00080000:3.0:1787698459.283094:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698459.283097:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698459.283105:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698459.283114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134598656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698459.283123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698459.283125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698459.283128:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698459.283143:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134598528:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698459.283145:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6361 00000100:00080000:3.0:1787698459.283150:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698459.283157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698459.283161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698459.283170:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134598784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698459.283177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698459.283180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698459.283182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698459.283184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6361 00000100:00080000:3.0:1787698459.283187:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698459.283189:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698459.283192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698459.283200:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134598912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698459.283206:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698459.283208:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698459.283209:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698459.283212:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6361 00000100:00080000:3.0:1787698459.283215:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698459.283220:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698459.283221:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698459.283228:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698459.283236:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134599040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698459.283242:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698459.283245:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698459.283247:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698459.283250:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6361 00000100:00080000:3.0:1787698459.283252:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698459.283254:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698459.283257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698459.283266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134599168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698459.283274:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698459/real 1787698459] req@ffff9d85c0fb2a00 x1874531134598528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698459.283274:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698459.283281:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698459.283284:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698459.283292:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6361 00000100:00080000:3.0:1787698459.283296:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698459.283301:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698459.283305:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698459.283315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134598528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698459.283315:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134599296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698459.283351:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698459.283358:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698459.283363:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134598528:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698459.283387:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598656:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698459.283409:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698459.283423:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698459/real 1787698459] req@ffff9d85c0fb2680 x1874531134598656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698459.283437:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134598656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698459.283446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698459.283450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698459.283454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698459.283460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698459.283481:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698459.283494:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698459/real 1787698459] req@ffff9d85c0fb0e00 x1874531134598784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698459.283508:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134598784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698459.283516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698459.283519:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698459.283523:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698459.283529:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698459.283551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698459.283562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698459/real 1787698459] req@ffff9d85c0fb0000 x1874531134598912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698459.283574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134598912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698459.283582:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698459.283585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698459.283589:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134598912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698459.283595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599040:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698459.283616:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698459.283627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698459/real 1787698459] req@ffff9d85c0fb3800 x1874531134599040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698459.283639:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134599040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698459.283648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698459.283651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698459.283654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599040:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698459.283660:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698459.283679:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698459.283690:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698459/real 1787698459] req@ffff9d85c0fb2d80 x1874531134599168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698459.283702:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134599168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698459.283710:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698459.283714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698459.283717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698459.283723:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698459.283743:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698459.283754:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698459/real 1787698459] req@ffff9d85c0fb0380 x1874531134599296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698459.283766:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134599296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698459.283774:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698459.283777:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698459.283781:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599296:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698459.858988:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698460.883058:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698461.907061:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698462.930995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698463.955047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698464.403014:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698464.403030:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698464.403033:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698464.403066:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6366 00000100:00080000:2.0:1787698464.403079:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698464.403086:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698464.403129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134599424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698464.403153:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698464.403156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698464.403157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698464.403160:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6366 00000100:00080000:2.0:1787698464.403162:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698464.403164:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698464.403172:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698464.403180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134599552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698464.403191:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698464.403194:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698464.403196:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698464.403201:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6366 00000100:00080000:2.0:1787698464.403208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698464.403212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698464.403218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698464.403233:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134599680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698464.403247:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698464.403250:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134599424:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698464.403252:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698464.403255:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698464.403267:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6366 00000100:00080000:2.0:1787698464.403272:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698464.403277:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698464.403281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698464.403297:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134599808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698464.403306:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698464.403310:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698464.403312:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698464.403329:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6366 00000800:00000100:1.0:1787698464.403336:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698464.403339:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698464.403345:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698464.403350:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698464.403362:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134599936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698464.403372:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698464.403375:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698464.403377:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698464.403380:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6366 00000100:00080000:2.0:1787698464.403382:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698464.403384:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698464.403387:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698464.403397:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134600064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698464.403405:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698464/real 1787698464] req@ffff9d85babd5500 x1874531134599424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698464.403406:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698464.403421:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698464.403426:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698464.403438:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6366 00000100:00080000:2.0:1787698464.403445:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698464.403451:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698464.403456:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698464.403466:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134600192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698464.403473:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134599424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698464.403487:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698464.403493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698464.403499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134599424:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698464.403508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134599552:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698464.403533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698464.403546:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698464/real 1787698464] req@ffff9d85babd6d80 x1874531134599552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698464.403559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134599552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698464.403568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698464.403572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698464.403575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134599552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698464.403600:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134599680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698464.403621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698464.403633:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698464/real 1787698464] req@ffff9d85babd7800 x1874531134599680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698464.403645:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134599680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698464.403654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698464.403658:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698464.403661:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134599680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698464.403667:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134599808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698464.403686:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698464.403698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698464/real 1787698464] req@ffff9d85babd6680 x1874531134599808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698464.403709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134599808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698464.403717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698464.403721:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698464.403724:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134599808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698464.403729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599936:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698464.403747:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698464.403758:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698464/real 1787698464] req@ffff9d85babd6a00 x1874531134599936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698464.403770:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134599936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698464.403778:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698464.403782:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698464.403785:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134599936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698464.403791:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698464.403809:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698464.403820:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698464/real 1787698464] req@ffff9d85babd6300 x1874531134600064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698464.403831:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134600064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698464.403839:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698464.403842:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698464.403845:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698464.403909:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698464.403934:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698464.403947:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698464/real 1787698464] req@ffff9d85babd4380 x1874531134600192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698464.403960:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134600192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698464.403969:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698464.403973:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698464.403976:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600192:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698464.979018:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698466.003092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698467.027064:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698468.051100:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698469.075096:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698469.523040:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698469.523051:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698469.523055:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698469.523067:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6371 00000100:00080000:2.0:1787698469.523080:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698469.523089:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698469.523144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134600320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698469.523171:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698469.523175:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698469.523176:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698469.523180:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6371 00000100:00080000:2.0:1787698469.523185:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698469.523189:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698469.523200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698469.523213:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134600448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698469.523222:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134600320:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698469.523222:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698469.523225:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698469.523227:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698469.523230:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6371 00000100:00080000:2.0:1787698469.523233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698469.523237:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698469.523242:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698469.523250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134600576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698469.523258:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698469.523261:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698469.523263:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698469.523266:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6371 00000100:00080000:2.0:1787698469.523269:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698469.523272:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698469.523276:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698469.523283:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134600704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698469.523290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698469.523292:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698469.523294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698469.523297:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6371 00000100:00080000:2.0:1787698469.523300:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698469.523304:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698469.523305:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698469.523308:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698469.523316:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134600832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698469.523323:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698469.523326:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698469.523328:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698469.523331:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6371 00000100:00080000:2.0:1787698469.523334:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698469.523337:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698469.523341:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698469.523349:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134600960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698469.523357:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698469.523358:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698469/real 1787698469] req@ffff9d85babd4000 x1874531134600320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698469.523360:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698469.523361:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698469.523365:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6371 00000100:00100000:1.0:1787698469.523376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134600320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698469.523378:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698469.523381:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698469.523385:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698469.523386:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698469.523390:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698469.523395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134600320:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698469.523395:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134601088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698469.523422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134600448:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698469.523442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698469.523455:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698469/real 1787698469] req@ffff9d85babd5c00 x1874531134600448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698469.523463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134600448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698469.523469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698469.523471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698469.523473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134600448:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698469.523477:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134600576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698469.523490:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698469.523497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698469/real 1787698469] req@ffff9d85babd7480 x1874531134600576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698469.523503:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134600576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698469.523508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698469.523509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698469.523511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134600576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698469.523514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134600704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698469.523523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698469.523528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698469/real 1787698469] req@ffff9d85babd4700 x1874531134600704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698469.523535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134600704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698469.523539:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698469.523541:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698469.523542:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134600704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698469.523545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600832:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698469.523554:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698469.523559:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698469/real 1787698469] req@ffff9d85babd5f80 x1874531134600832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698469.523564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134600832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698469.523570:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698469.523571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698469.523573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698469.523576:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698469.523584:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698469.523589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698469/real 1787698469] req@ffff9d85babd7100 x1874531134600960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698469.523594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134600960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698469.523599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698469.523600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698469.523602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134600960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698469.523604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698469.523612:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698469.523617:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698469/real 1787698469] req@ffff9d85babd4a80 x1874531134601088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698469.523622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134601088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698469.523627:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698469.523629:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698469.523631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601088:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698470.099069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698471.123055:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698472.147091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698473.171003:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698474.195075:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698474.643039:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698474.643053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698474.643057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698474.643070:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6376 00000100:00080000:3.0:1787698474.643084:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698474.643093:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698474.643137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134601216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698474.643161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698474.643164:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698474.643165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698474.643169:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6376 00000100:00080000:3.0:1787698474.643172:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698474.643174:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698474.643183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698474.643192:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134601344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698474.643198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698474.643207:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698474.643208:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698474.643213:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134601216:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698474.643214:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6376 00000100:00080000:3.0:1787698474.643220:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698474.643223:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698474.643227:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698474.643235:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134601472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698474.643241:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698474.643243:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698474.643244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698474.643248:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6376 00000100:00080000:3.0:1787698474.643250:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698474.643252:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698474.643255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698474.643262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134601600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698474.643267:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698474.643270:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698474.643272:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698474.643274:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6376 00000100:00080000:3.0:1787698474.643276:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698474.643279:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698474.643281:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698474.643283:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698474.643291:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134601728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698474.643298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698474.643301:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698474.643302:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698474.643305:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6376 00000100:00080000:3.0:1787698474.643307:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698474.643309:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698474.643312:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698474.643319:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134601856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698474.643323:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698474.643326:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698474.643327:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698474.643329:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6376 00000100:00080000:3.0:1787698474.643331:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698474.643338:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698474.643342:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698474.643350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698474/real 1787698474] req@ffff9d85c0fb3b80 x1874531134601216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698474.643350:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134601984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698474.643383:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134601216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698474.643391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698474.643394:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698474.643398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134601216:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698474.643405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134601344:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698474.643427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698474.643440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698474/real 1787698474] req@ffff9d85c0fb0a80 x1874531134601344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698474.643453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134601344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698474.643460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698474.643464:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698474.643467:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134601344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698474.643493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134601472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698474.643510:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698474.643518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698474/real 1787698474] req@ffff9d85c0fb1c00 x1874531134601472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698474.643539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134601472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698474.643544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698474.643546:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698474.643548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134601472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698474.643552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134601600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698474.643565:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698474.643575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698474/real 1787698474] req@ffff9d85c0fb1f80 x1874531134601600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698474.643586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134601600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698474.643594:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698474.643597:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698474.643600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134601600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698474.643605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601728:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698474.643621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698474.643628:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698474/real 1787698474] req@ffff9d85c0fb3100 x1874531134601728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698474.643634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134601728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698474.643640:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698474.643642:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698474.643644:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698474.643646:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698474.643655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698474.643660:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698474/real 1787698474] req@ffff9d85c0fb1880 x1874531134601856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698474.643666:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134601856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698474.643674:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698474.643677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698474.643680:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698474.643685:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698474.643704:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698474.643715:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698474/real 1787698474] req@ffff9d85c0fb1500 x1874531134601984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698474.643726:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134601984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698474.643734:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698474.643737:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698474.643741:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134601984:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698475.219015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698476.243037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698477.266996:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698478.291088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698479.315048:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698479.762965:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698479.762976:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698479.762980:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698479.762990:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6381 00000100:00080000:3.0:1787698479.763000:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698479.763006:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698479.763041:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134602112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698479.763069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698479.763072:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698479.763074:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698479.763078:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6381 00000100:00080000:3.0:1787698479.763081:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698479.763085:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698479.763100:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698479.763105:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134602112:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698479.763116:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134602240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698479.763126:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698479.763129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698479.763130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698479.763134:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6381 00000100:00080000:3.0:1787698479.763137:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698479.763141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698479.763146:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698479.763163:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134602368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698479.763165:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698479.763175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698479.763179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698479.763181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698479.763185:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6381 00000100:00080000:3.0:1787698479.763189:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698479.763194:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698479.763198:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698479.763209:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698479/real 1787698479] req@ffff9d85c0fb0700 x1874531134602112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698479.763213:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134602496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698479.763230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698479.763234:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787698479.763237:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134602112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698479.763237:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698479.763245:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6381 00000100:00080000:1.0:1787698479.763249:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698479.763252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698479.763254:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698479.763262:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134602112:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698479.763262:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698479.763271:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698479.763284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134602240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698479.763285:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134602624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698479.763299:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698479.763302:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787698479.763305:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698479.763307:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698479.763314:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6381 00000100:00100000:1.0:1787698479.763319:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698479/real 1787698479] req@ffff9d85c0fb3480 x1874531134602240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698479.763320:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698479.763329:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698479.763333:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134602240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698479.763336:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698479.763341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698479.763344:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698479.763347:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134602240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698479.763360:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134602368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698479.763362:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134602752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698479.763373:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698479.763376:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698479.763380:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698479.763383:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698479.763388:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698479/real 1787698479] req@ffff9d85c0fb2300 x1874531134602368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698479.763389:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6381 00000100:00080000:3.0:1787698479.763396:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698479.763403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134602368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698479.763404:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698479.763413:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698479.763414:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698479.763420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698479.763423:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134602368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698479.763430:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134602496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698479.763431:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134602880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698479.763442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698479.763451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698479/real 1787698479] req@ffff9d85c0fb1500 x1874531134602496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698479.763460:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134602496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698479.763466:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698479.763467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698479.763469:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134602496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698479.763477:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134602624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698479.763486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698479.763492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698479/real 1787698479] req@ffff9d85c0fb1880 x1874531134602624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698479.763497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134602624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698479.763502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698479.763504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698479.763506:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134602624:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698479.763508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134602752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698479.763516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698479.763521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698479/real 1787698479] req@ffff9d85c0fb3100 x1874531134602752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698479.763527:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134602752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698479.763532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698479.763533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698479.763535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134602752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698479.763538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134602880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698479.763545:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698479.763550:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698479/real 1787698479] req@ffff9d85c0fb1f80 x1874531134602880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698479.763556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134602880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698479.763560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698479.763562:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698479.763564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134602880:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698480.339045:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698481.363072:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698482.386997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698483.411071:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698484.434979:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698484.882897:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698484.882904:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698484.882907:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698484.882914:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6386 00000100:00080000:3.0:1787698484.882922:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698484.882927:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698484.882955:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134603008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698484.882974:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698484.882977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698484.882978:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698484.882981:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6386 00000100:00080000:3.0:1787698484.882984:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698484.882988:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698484.882996:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698484.883009:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134603136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698484.883011:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134603008:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698484.883025:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698484.883028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698484.883031:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698484.883034:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6386 00000100:00080000:3.0:1787698484.883037:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698484.883040:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698484.883044:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698484.883054:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134603264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698484.883060:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698484.883062:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698484.883063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698484.883065:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6386 00000800:00000100:1.0:1787698484.883069:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698484.883070:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698484.883075:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698484.883078:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698484.883085:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134603392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698484.883095:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698484.883098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698484.883099:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698484.883102:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6386 00000100:00080000:3.0:1787698484.883106:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698484.883110:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698484.883112:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698484/real 1787698484] req@ffff9d85c0fb1c00 x1874531134603008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698484.883116:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698484.883132:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134603008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698484.883136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134603520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698484.883145:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.883151:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698484.883152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698484.883155:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134603008:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698484.883159:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698484.883170:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698484.883175:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6386 00000100:00080000:3.0:1787698484.883179:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698484.883182:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134603136:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698484.883186:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698484.883194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698484.883207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134603648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698484.883215:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698484.883224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698484.883228:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698484.883230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698484.883234:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698484/real 1787698484] req@ffff9d85c0fb0a80 x1874531134603136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698484.883237:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6386 00000100:00080000:3.0:1787698484.883245:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698484.883251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698484.883255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698484.883256:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134603136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698484.883267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134603776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698484.883268:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.883277:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698484.883284:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134603136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698484.883291:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134603264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698484.883311:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698484.883324:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698484/real 1787698484] req@ffff9d85c0fb3b80 x1874531134603264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.883334:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134603264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.883341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.883345:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698484.883348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134603264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698484.883354:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134603392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698484.883371:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698484.883381:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698484/real 1787698484] req@ffff9d85c0fb0380 x1874531134603392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.883392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134603392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.883399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.883402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698484.883405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134603392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698484.883419:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134603520:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698484.883440:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698484.883450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698484/real 1787698484] req@ffff9d85c0fb2d80 x1874531134603520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.883460:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134603520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.883467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.883469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698484.883472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134603520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698484.883488:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134603648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698484.883503:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698484.883511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698484/real 1787698484] req@ffff9d85c0fb3800 x1874531134603648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.883522:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134603648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.883529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.883532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698484.883534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134603648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698484.883540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134603776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698484.883553:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698484.883562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698484/real 1787698484] req@ffff9d85c0fb0000 x1874531134603776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.883572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134603776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.883579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.883582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698484.883585:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134603776:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698485.459025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698486.482981:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698487.507046:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698488.530993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698489.555003:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698490.002984:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698490.002993:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698490.002996:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698490.003003:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6391 00000100:00080000:1.0:1787698490.003012:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698490.003017:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698490.003051:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134603904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698490.003082:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698490.003085:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698490.003086:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698490.003089:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6391 00000100:00080000:1.0:1787698490.003092:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698490.003094:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698490.003102:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698490.003114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134604032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698490.003122:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134603904:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698490.003126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698490.003132:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698490.003134:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698490.003138:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6391 00000100:00080000:1.0:1787698490.003142:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698490.003146:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698490.003152:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698490.003162:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134604160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698490.003168:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698490.003170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698490.003172:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698490.003174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6391 00000100:00080000:1.0:1787698490.003178:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698490.003180:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698490.003182:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698490.003188:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698490.003191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134604288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698490.003198:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698490.003200:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698490.003202:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698490.003205:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6391 00000100:00080000:1.0:1787698490.003209:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698490.003211:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698490.003213:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698490.003219:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134604416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698490.003224:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698490.003227:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698490.003228:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698490.003233:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6391 00000100:00100000:0.0:1787698490.003234:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698490/real 1787698490] req@ffff9d85b9d2e680 x1874531134603904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698490.003235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698490.003240:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698490.003245:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698490.003254:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134604544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698490.003260:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698490.003262:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134603904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698490.003263:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698490.003265:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698490.003267:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6391 00000100:00080000:1.0:1787698490.003269:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698490.003271:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698490.003276:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698490.003287:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698490.003287:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134604672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698490.003297:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698490.003301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134603904:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698490.003324:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604032:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698490.003341:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698490.003352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698490/real 1787698490] req@ffff9d85b9d2d500 x1874531134604032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698490.003363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134604032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698490.003371:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698490.003374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698490.003377:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698490.003383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698490.003399:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698490.003406:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698490/real 1787698490] req@ffff9d85b9d2ed80 x1874531134604160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698490.003414:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134604160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698490.003418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698490.003420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698490.003422:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698490.003425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698490.003434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698490.003439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698490/real 1787698490] req@ffff9d85b9d2c700 x1874531134604288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698490.003446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134604288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698490.003457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698490.003459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698490.003461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698490.003464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134604416:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698490.003474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698490.003479:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698490/real 1787698490] req@ffff9d85b9d2c380 x1874531134604416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698490.003486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134604416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698490.003494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698490.003496:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698490.003498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134604416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698490.003501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134604544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698490.003509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698490.003513:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698490/real 1787698490] req@ffff9d85b9d2dc00 x1874531134604544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698490.003519:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134604544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698490.003523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698490.003525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698490.003526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134604544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698490.003529:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134604672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698490.003537:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698490.003541:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698490/real 1787698490] req@ffff9d85b9d2e300 x1874531134604672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698490.003547:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134604672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698490.003551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698490.003552:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698490.003554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134604672:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698490.579002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698491.603028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698492.626984:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698493.651026:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698494.675030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698495.122934:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698495.122943:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698495.122947:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698495.122958:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6396 00000100:00080000:3.0:1787698495.122971:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698495.122979:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698495.123020:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134604800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698495.123048:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698495.123052:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698495.123054:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698495.123058:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6396 00000100:00080000:3.0:1787698495.123062:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698495.123066:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698495.123076:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698495.123090:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134604928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698495.123106:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698495.123110:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134604800:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698495.123111:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698495.123115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698495.123122:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6396 00000100:00080000:3.0:1787698495.123125:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698495.123129:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698495.123135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698495.123146:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134605056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698495.123154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698495.123158:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698495.123161:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698495.123164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6396 00000100:00080000:3.0:1787698495.123168:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698495.123172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698495.123177:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698495.123185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698495.123191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134605184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698495.123202:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698495.123205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698495.123207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698495.123219:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6396 00000100:00100000:0.0:1787698495.123222:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698495/real 1787698495] req@ffff9d85c0fb0e00 x1874531134604800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698495.123226:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698495.123234:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698495.123239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698495.123248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134605312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698495.123253:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134604800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698495.123257:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698495.123264:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698495.123266:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698495.123271:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6396 00000100:00080000:0.0:1787698495.123272:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698495.123274:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698495.123280:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698495.123280:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698495.123286:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698495.123289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134604800:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698495.123296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134605440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698495.123303:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698495.123305:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698495.123307:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698495.123309:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6396 00000100:00080000:3.0:1787698495.123311:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698495.123313:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604928:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698495.123316:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698495.123321:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698495.123329:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134605568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698495.123338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698495.123352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698495/real 1787698495] req@ffff9d85c0fb2680 x1874531134604928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698495.123364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134604928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698495.123369:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698495.123371:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698495.123373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134604928:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698495.123376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698495.123387:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698495.123393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698495/real 1787698495] req@ffff9d85c0fb2a00 x1874531134605056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698495.123399:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134605056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698495.123404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698495.123405:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698495.123407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698495.123409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698495.123418:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698495.123423:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698495/real 1787698495] req@ffff9d85c0fb0000 x1874531134605184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698495.123429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134605184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698495.123434:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698495.123435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698495.123436:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698495.123439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134605312:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698495.123447:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698495.123452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698495/real 1787698495] req@ffff9d85c0fb3800 x1874531134605312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698495.123457:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134605312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698495.123469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698495.123471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698495.123473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134605312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698495.123476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134605440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698495.123486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698495.123492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698495/real 1787698495] req@ffff9d85c0fb2d80 x1874531134605440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698495.123498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134605440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698495.123503:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698495.123504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698495.123506:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134605440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698495.123516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134605568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698495.123541:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698495.123547:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698495/real 1787698495] req@ffff9d85c0fb0380 x1874531134605568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698495.123554:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134605568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698495.123559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698495.123561:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698495.123563:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134605568:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698495.699092:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698496.722975:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698497.747056:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698498.771073:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698499.795044:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698500.242945:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698500.242955:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698500.242960:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698500.242970:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6402 00000100:00080000:3.0:1787698500.242982:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698500.242990:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698500.243053:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134605696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698500.243088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698500.243092:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698500.243094:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698500.243098:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6402 00000100:00080000:3.0:1787698500.243102:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698500.243108:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698500.243120:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698500.243132:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134605696:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698500.243137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698500.243149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698500.243152:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698500.243154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698500.243158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6402 00000100:00080000:3.0:1787698500.243161:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698500.243166:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698500.243173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698500.243184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134605952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698500.243193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698500.243196:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698500.243198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698500.243203:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698500.243206:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6402 00000100:00080000:3.0:1787698500.243212:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698500.243216:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698500.243222:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698500.243236:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134606080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698500.243246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698500.243250:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698500/real 1787698500] req@ffff9d85c0fb3b80 x1874531134605696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698500.243252:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698500.243257:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698500.243265:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6402 00000100:00080000:3.0:1787698500.243269:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698500.243271:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134605696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698500.243277:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698500.243283:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698500.243285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698500.243288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698500.243292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134605696:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698500.243303:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134606208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698500.243315:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698500.243316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605824:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698500.243321:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698500.243324:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698500.243328:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6402 00000100:00080000:3.0:1787698500.243331:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698500.243334:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698500.243338:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698500.243348:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698500.243356:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134606336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698500.243364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698500/real 1787698500] req@ffff9d85c0fb0a80 x1874531134605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698500.243365:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698500.243370:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698500.243373:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698500.243380:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6402 00000100:00080000:3.0:1787698500.243385:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698500.243388:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698500.243391:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698500.243392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698500.243400:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134606464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698500.243408:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698500.243422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698500.243426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605824:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698500.243441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698500.243459:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698500.243468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698500/real 1787698500] req@ffff9d85c0fb1c00 x1874531134605952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698500.243475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134605952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698500.243480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698500.243482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698500.243484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134605952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698500.243487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698500.243496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698500.243501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698500/real 1787698500] req@ffff9d85c0fb1f80 x1874531134606080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698500.243508:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134606080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698500.243512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698500.243514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698500.243515:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698500.243525:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134606208:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698500.243534:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698500.243539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698500/real 1787698500] req@ffff9d85c0fb3100 x1874531134606208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698500.243544:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134606208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698500.243550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698500.243551:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698500.243553:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134606208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698500.243556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134606336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698500.243564:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698500.243568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698500/real 1787698500] req@ffff9d85c0fb1880 x1874531134606336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698500.243575:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134606336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698500.243580:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698500.243582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698500.243583:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134606336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698500.243586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134606464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698500.243593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698500.243598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698500/real 1787698500] req@ffff9d85c0fb1500 x1874531134606464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698500.243603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134606464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698500.243608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698500.243609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698500.243612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134606464:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698500.819082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698501.843035:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698502.867034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698503.890991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698504.915112:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698505.363024:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698505.363039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698505.363044:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698505.363057:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6407 00000100:00080000:2.0:1787698505.363071:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698505.363080:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698505.363122:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134606592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698505.363150:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698505.363153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698505.363155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698505.363160:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6407 00000100:00080000:2.0:1787698505.363164:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698505.363168:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698505.363180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698505.363194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134606720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698505.363202:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698505.363205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698505.363207:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698505.363210:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134606592:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698505.363212:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6407 00000100:00080000:2.0:1787698505.363217:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698505.363221:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698505.363225:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698505.363232:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134606848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698505.363237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698505.363239:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698505.363241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698505.363244:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6407 00000100:00080000:2.0:1787698505.363247:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698505.363250:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698505.363255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698505.363269:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134606976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698505.363284:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698505.363287:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698505.363289:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698505.363293:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6407 00000100:00080000:2.0:1787698505.363296:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698505.363300:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698505.363306:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698505.363307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698505.363320:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698505.363329:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698505.363333:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698505.363335:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698505.363339:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6407 00000100:00080000:2.0:1787698505.363368:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698505.363374:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698505.363378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698505/real 1787698505] req@ffff9d85babd7b80 x1874531134606592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698505.363381:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698505.363399:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134607232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698505.363405:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134606592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698505.363414:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698505.363421:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698505.363424:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698505.363425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698505.363430:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6407 00000100:00080000:0.0:1787698505.363433:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698505.363437:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698505.363441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134606592:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698505.363445:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698505.363453:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698505.363471:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698505.363473:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134607360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698505.363498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698505.363512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698505/real 1787698505] req@ffff9d85babd4e00 x1874531134606720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698505.363531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134606720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698505.363536:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698505.363538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698505.363541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698505.363544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698505.363558:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698505.363565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698505/real 1787698505] req@ffff9d85babd5180 x1874531134606848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698505.363572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134606848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698505.363576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698505.363578:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698505.363579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698505.363582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698505.363592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698505.363597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698505/real 1787698505] req@ffff9d85babd4a80 x1874531134606976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698505.363603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134606976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698505.363608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698505.363609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698505.363611:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134606976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698505.363613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134607104:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698505.363622:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698505.363627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698505/real 1787698505] req@ffff9d85babd7100 x1874531134607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698505.363633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698505.363639:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698505.363641:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698505.363643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134607104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698505.363645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134607232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698505.363654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698505.363658:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698505/real 1787698505] req@ffff9d85babd5f80 x1874531134607232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698505.363664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134607232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698505.363668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698505.363670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698505.363672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134607232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698505.363683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134607360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698505.363692:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698505.363697:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698505/real 1787698505] req@ffff9d85babd4700 x1874531134607360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698505.363704:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134607360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698505.363708:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698505.363710:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698505.363712:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134607360:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698505.939066:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698506.963064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698507.987034:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698509.011044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698510.034991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698510.482948:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698510.482958:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698510.482963:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698510.482973:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6412 00000100:00080000:2.0:1787698510.482985:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698510.482992:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698510.483033:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134607488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698510.483058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698510.483060:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698510.483062:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698510.483065:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6412 00000100:00080000:2.0:1787698510.483067:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698510.483070:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698510.483079:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698510.483087:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134607616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698510.483096:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134607488:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698510.483096:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698510.483102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698510.483105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698510.483109:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6412 00000100:00080000:2.0:1787698510.483111:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698510.483113:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698510.483116:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698510.483124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134607744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698510.483129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698510.483131:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698510.483133:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698510.483135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6412 00000100:00080000:2.0:1787698510.483139:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698510.483143:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698510.483148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698510.483158:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134607872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698510.483168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698510.483171:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698510.483173:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698510.483177:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6412 00000100:00080000:2.0:1787698510.483181:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698510.483186:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698510.483191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698510.483205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134608000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698510.483218:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698510.483222:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134607616:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698510.483224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698510.483231:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698510.483236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6412 00000100:00080000:2.0:1787698510.483241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698510.483245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698510.483249:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134607744:192.168.201.137@tcp:8:kworker.0 00000008:00100000:2.0:1787698510.483251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698510.483293:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134607872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698510.483307:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698510.483353:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134608128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698510.483367:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698510.483370:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698510.483372:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698510.483378:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6412 00000100:00080000:2.0:1787698510.483381:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698510.483385:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698510.483393:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698510.483397:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698510.483404:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134608256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698510.483427:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608256:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698511.059043:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698512.083001:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698513.107033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698513.555134:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:1.0:1787698513.555178:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698513.555194:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698513.555269:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698513.555278:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698513.555288:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698513.555297:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:1.0:1787698513.555323:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698513.555328:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698510/real 1787698513] req@ffff9d85babd5c00 x1874531134607616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698513.555343:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698513.555363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134607616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698513.555380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698513.555386:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698513.555392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134607616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698513.555401:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698510/real 1787698513] req@ffff9d85babd4000 x1874531134607744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698513.555432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134607744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698513.555445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698513.555449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698513.555452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134607744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698513.555459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698510/real 1787698513] req@ffff9d85babd4380 x1874531134607872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698513.555470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134607872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698513.555477:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698513.555480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698513.555484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134607872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698513.555491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698510/real 1787698513] req@ffff9d85babd6300 x1874531134608000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698513.555502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134608000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698513.555510:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698513.555513:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698513.555516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698513.555523:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698510/real 1787698513] req@ffff9d85babd7480 x1874531134607488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698513.555536:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134607488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698513.555545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698513.555548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698513.555552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134607488:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698513.555583:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698510/real 1787698513] req@ffff9d85babd6680 x1874531134608256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698513.555597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134608256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698513.555605:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698513.555609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698513.555613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698513.555620:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698510/real 1787698513] req@ffff9d85babd6a00 x1874531134608128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698513.555631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134608128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698513.555638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698513.555642:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698513.555645:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608128:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698514.131015:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698515.154996:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698515.602936:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698515.602945:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698515.602947:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698515.602955:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6417 00000100:00080000:2.0:1787698515.602965:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698515.602971:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698515.603002:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134608384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698515.603024:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698515.603027:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698515.603028:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698515.603031:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6417 00000100:00080000:2.0:1787698515.603034:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698515.603036:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698515.603042:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698515.603050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134608512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698515.603058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698515.603061:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698515.603063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698515.603066:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6417 00000100:00080000:2.0:1787698515.603068:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698515.603071:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698515.603074:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698515.603081:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134608640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698515.603088:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698515.603091:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698515.603092:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134608384:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698515.603093:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698515.603100:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6417 00000100:00080000:2.0:1787698515.603105:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698515.603108:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698515.603110:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698515.603122:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134608768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698515.603132:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698515.603135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698515.603138:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698515.603142:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6417 00000100:00080000:2.0:1787698515.603146:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698515.603150:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698515.603155:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698515.603185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698515.603202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134608896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698515.603213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698515.603217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698515.603219:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698515.603223:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6417 00000100:00080000:2.0:1787698515.603228:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698515.603233:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698515/real 1787698515] req@ffff9d85babd7800 x1874531134608384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698515.603243:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698515.603253:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698515.603262:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134608384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698515.603276:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698515.603277:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134609024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698515.603286:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698515.603296:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698515.603298:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134608384:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698515.603304:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698515.603308:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698515.603312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134608512:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698515.603325:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6417 00000100:00080000:2.0:1787698515.603330:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698515.603335:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698515.603341:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698515.603345:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698515.603359:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134609152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698515.603366:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698515/real 1787698515] req@ffff9d85babd6d80 x1874531134608512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698515.603386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134608512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698515.603396:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698515.603400:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698515.603403:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134608512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698515.603439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134608640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698515.603465:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698515.603476:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698515/real 1787698515] req@ffff9d85babd5500 x1874531134608640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698515.603489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134608640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698515.603502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698515.603505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698515.603508:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134608640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698515.603515:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134608768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698515.603534:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698515.603546:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698515/real 1787698515] req@ffff9d85babd6a00 x1874531134608768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698515.603557:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134608768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698515.603565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698515.603568:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698515.603571:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134608768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698515.603577:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608896:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698515.603596:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698515.603608:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698515/real 1787698515] req@ffff9d85babd6680 x1874531134608896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698515.603619:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134608896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698515.603628:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698515.603631:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698515.603635:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134608896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698515.603642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698515.603662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698515.603673:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698515/real 1787698515] req@ffff9d85babd7480 x1874531134609024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698515.603684:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134609024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698515.603692:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698515.603695:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698515.603698:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698515.603704:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698515.603723:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698515.603734:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698515/real 1787698515] req@ffff9d85babd6300 x1874531134609152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698515.603745:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134609152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698515.603752:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698515.603756:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698515.603759:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609152:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698516.179062:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698517.203049:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698518.227001:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698519.251063:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698520.275071:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698520.723055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698520.723070:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698520.723076:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698520.723104:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6422 00000100:00080000:3.0:1787698520.723119:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698520.723129:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698520.723180:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134609280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698520.723213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698520.723218:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698520.723220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698520.723225:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6422 00000100:00080000:3.0:1787698520.723230:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698520.723235:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698520.723259:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698520.723260:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134609280:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698520.723280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134609408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698520.723294:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698520.723298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698520.723300:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698520.723305:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6422 00000100:00080000:3.0:1787698520.723310:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698520.723315:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698520.723321:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698520.723337:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134609536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698520.723338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698520.723359:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698520.723364:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698520.723365:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698520.723370:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6422 00000100:00080000:3.0:1787698520.723374:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698520.723379:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698520.723385:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698520.723394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698520/real 1787698520] req@ffff9d85c0fb2300 x1874531134609280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698520.723402:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134609664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698520.723420:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698520.723426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134609280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698520.723427:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698520.723432:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698520.723443:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6422 00000100:00080000:0.0:1787698520.723445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698520.723449:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698520.723451:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698520.723460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134609280:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698520.723460:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698520.723471:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698520.723486:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134609792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698520.723491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134609408:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698520.723508:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698520.723513:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698520.723516:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698520.723519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698520.723525:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6422 00000100:00080000:3.0:1787698520.723532:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698520.723534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698520/real 1787698520] req@ffff9d85c0fb3480 x1874531134609408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698520.723541:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698520.723550:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134609408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698520.723550:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698520.723563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698520.723565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698520.723571:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134609408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698520.723573:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134609920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698520.723578:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134609536:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698520.723593:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698520.723598:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698520.723600:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698520.723602:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698520.723609:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6422 00000100:00100000:0.0:1787698520.723611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698520/real 1787698520] req@ffff9d85c0fb0700 x1874531134609536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698520.723615:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698520.723625:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698520.723630:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134609536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698520.723633:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698520.723652:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134610048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698520.723655:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698520.723665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698520.723675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134609536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698520.723683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134609664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698520.723702:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698520.723711:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698520/real 1787698520] req@ffff9d85c0fb1500 x1874531134609664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698520.723719:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134609664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698520.723725:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698520.723727:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698520.723729:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134609664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698520.723740:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609792:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698520.723760:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698520.723772:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698520/real 1787698520] req@ffff9d85c0fb1880 x1874531134609792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698520.723786:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134609792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698520.723794:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698520.723797:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698520.723801:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698520.723807:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698520.723827:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698520.723838:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698520/real 1787698520] req@ffff9d85c0fb3100 x1874531134609920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698520.723901:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134609920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698520.723910:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698520.723912:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698520.723915:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134609920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698520.723920:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698520.723937:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698520.723945:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698520/real 1787698520] req@ffff9d85c0fb1f80 x1874531134610048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698520.723953:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134610048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698520.723959:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698520.723961:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698520.723963:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610048:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698521.299090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698522.323077:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698523.347073:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698524.371107:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698525.394997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698525.843025:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698525.843036:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698525.843041:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698525.843051:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6427 00000100:00080000:1.0:1787698525.843064:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698525.843071:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698525.843117:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134610176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698525.843146:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698525.843149:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698525.843150:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698525.843154:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6427 00000100:00080000:1.0:1787698525.843156:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698525.843159:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698525.843169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698525.843181:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134610304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698525.843190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698525.843193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698525.843194:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698525.843199:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6427 00000100:00080000:1.0:1787698525.843202:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698525.843204:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134610176:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698525.843207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698525.843215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698525.843222:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134610432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698525.843228:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698525.843231:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698525.843233:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698525.843235:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6427 00000100:00080000:1.0:1787698525.843237:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698525.843239:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698525.843242:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698525.843251:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134610560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698525.843257:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698525.843259:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698525.843260:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698525.843262:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6427 00000100:00080000:1.0:1787698525.843275:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698525.843279:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698525.843282:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698525.843288:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698525.843292:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134610688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698525.843299:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698525.843303:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698525.843304:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698525.843308:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6427 00000100:00080000:1.0:1787698525.843311:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698525.843313:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698525.843316:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698525.843323:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134610816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698525.843329:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698525.843331:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698525.843332:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698525.843336:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6427 00000100:00080000:1.0:1787698525.843339:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698525.843343:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698525.843348:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698525.843359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698525/real 1787698525] req@ffff9d85b9d2f480 x1874531134610176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698525.843363:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134610944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698525.843399:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134610176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698525.843413:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698525.843417:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698525.843420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134610176:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698525.843444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134610304:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698525.843464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698525.843474:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698525/real 1787698525] req@ffff9d85b9d2fb80 x1874531134610304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698525.843488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134610304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698525.843497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698525.843501:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698525.843505:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134610304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698525.843512:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134610432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698525.843534:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698525.843543:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698525/real 1787698525] req@ffff9d85b9d2f800 x1874531134610432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698525.843552:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134610432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698525.843563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698525.843565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698525.843567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134610432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698525.843572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134610560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698525.843583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698525.843589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698525/real 1787698525] req@ffff9d85b9d2ca80 x1874531134610560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698525.843597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134610560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698525.843602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698525.843604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698525.843606:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134610560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698525.843609:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610688:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698525.843620:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698525.843626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698525/real 1787698525] req@ffff9d85b9d2c000 x1874531134610688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698525.843634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134610688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698525.843638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698525.843640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698525.843643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698525.843655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698525.843667:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698525.843674:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698525/real 1787698525] req@ffff9d85b9d2ce00 x1874531134610816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698525.843681:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134610816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698525.843686:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698525.843688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698525.843690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698525.843693:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698525.843704:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698525.843709:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698525/real 1787698525] req@ffff9d85b9d2e300 x1874531134610944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698525.843717:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134610944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698525.843722:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698525.843724:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698525.843726:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134610944:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698526.419092:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698527.443052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698528.467144:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698529.491061:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698530.515091:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698530.963001:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698530.963017:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698530.963022:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698530.963033:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6432 00000100:00080000:3.0:1787698530.963045:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698530.963053:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698530.963097:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134611072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698530.963123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698530.963126:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698530.963128:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698530.963131:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6432 00000100:00080000:3.0:1787698530.963133:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698530.963136:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698530.963145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698530.963154:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134611200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698530.963160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698530.963163:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698530.963164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698530.963167:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6432 00000100:00080000:3.0:1787698530.963169:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698530.963171:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698530.963175:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698530.963182:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134611072:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698530.963183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134611328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698530.963196:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698530.963199:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698530.963200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698530.963202:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6432 00000100:00080000:3.0:1787698530.963205:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698530.963207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698530.963210:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698530.963218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134611456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698530.963223:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698530.963225:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698530.963226:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698530.963229:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6432 00000100:00080000:3.0:1787698530.963231:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698530.963234:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698530.963237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698530.963243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134611584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698530.963248:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698530.963250:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698530.963252:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698530.963255:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6432 00000100:00080000:3.0:1787698530.963257:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698530.963259:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698530.963261:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698530.963269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134611712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698530.963271:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698530.963282:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698530.963285:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698530.963286:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698530.963289:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6432 00000100:00080000:3.0:1787698530.963290:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698530.963293:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698530.963295:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698530.963301:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134611840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698530.963328:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698530/real 1787698530] req@ffff9d85c0fb1c00 x1874531134611072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698530.963346:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134611072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698530.963356:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698530.963360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698530.963364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134611072:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698530.963371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134611200:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698530.963390:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698530.963400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698530/real 1787698530] req@ffff9d85c0fb0a80 x1874531134611200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698530.963418:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134611200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698530.963426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698530.963428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698530.963432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134611200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698530.963456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134611328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698530.963474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698530.963483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698530/real 1787698530] req@ffff9d85c0fb3b80 x1874531134611328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698530.963493:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134611328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698530.963505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698530.963508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698530.963511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134611328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698530.963517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134611456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698530.963531:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698530.963539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698530/real 1787698530] req@ffff9d85c0fb0380 x1874531134611456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698530.963549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134611456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698530.963556:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698530.963559:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698530.963562:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134611456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698530.963568:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134611584:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698530.963581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698530.963589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698530/real 1787698530] req@ffff9d85c0fb2d80 x1874531134611584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698530.963599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134611584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698530.963606:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698530.963609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698530.963613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134611584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698530.963618:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134611712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698530.963631:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698530.963640:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698530/real 1787698530] req@ffff9d85c0fb3800 x1874531134611712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698530.963649:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134611712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698530.963656:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698530.963659:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698530.963662:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134611712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698530.963667:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134611840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698530.963680:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698530.963688:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698530/real 1787698530] req@ffff9d85c0fb0000 x1874531134611840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698530.963698:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134611840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698530.963705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698530.963708:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698530.963711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134611840:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698531.539031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698532.563066:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698533.587021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698534.611069:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698535.635060:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698536.082906:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698536.082916:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698536.082921:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698536.082932:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6437 00000100:00080000:3.0:1787698536.082943:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698536.082951:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698536.082995:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134611968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698536.083022:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698536.083025:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698536.083026:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698536.083029:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6437 00000100:00080000:3.0:1787698536.083031:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698536.083034:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698536.083044:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698536.083052:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134612096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698536.083058:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698536.083061:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698536.083062:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698536.083065:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6437 00000100:00080000:3.0:1787698536.083066:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698536.083068:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698536.083072:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698536.083078:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134611968:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698536.083079:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134612224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698536.083093:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698536.083095:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698536.083096:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698536.083099:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6437 00000100:00080000:3.0:1787698536.083100:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698536.083102:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698536.083119:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698536.083134:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134612352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698536.083143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698536.083147:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698536.083149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698536.083153:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6437 00000100:00080000:3.0:1787698536.083157:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698536.083162:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698536.083166:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787698536.083169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698536.083183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134612480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698536.083190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698536.083192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698536.083194:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698536.083197:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6437 00000100:00080000:3.0:1787698536.083199:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698536.083202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698536.083205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698536.083212:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134612608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698536.083217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698536.083220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698536.083222:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698536.083225:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698536/real 1787698536] req@ffff9d85c0fb2a00 x1874531134611968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698536.083226:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6437 00000100:00080000:3.0:1787698536.083230:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698536.083235:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698536.083241:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698536.083248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134612736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698536.083270:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134611968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698536.083282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698536.083287:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698536.083292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134611968:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698536.083299:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612096:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698536.083321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698536.083333:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698536/real 1787698536] req@ffff9d85c0fb2680 x1874531134612096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698536.083344:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134612096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698536.083353:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698536.083356:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698536.083359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698536.083385:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698536.083403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698536.083413:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698536/real 1787698536] req@ffff9d85c0fb0e00 x1874531134612224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698536.083424:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134612224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698536.083437:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698536.083440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698536.083443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698536.083449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698536.083464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698536.083479:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698536/real 1787698536] req@ffff9d85c0fb0000 x1874531134612352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698536.083489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134612352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698536.083497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698536.083499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698536.083502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698536.083507:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134612480:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698536.083525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698536.083534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698536/real 1787698536] req@ffff9d85c0fb3800 x1874531134612480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698536.083545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134612480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698536.083552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698536.083555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698536.083559:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134612480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698536.083565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134612608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698536.083580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698536.083589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698536/real 1787698536] req@ffff9d85c0fb2d80 x1874531134612608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698536.083598:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134612608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698536.083605:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698536.083608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698536.083611:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134612608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698536.083616:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134612736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698536.083631:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698536.083639:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698536/real 1787698536] req@ffff9d85c0fb0380 x1874531134612736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698536.083649:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134612736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698536.083656:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698536.083658:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698536.083661:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134612736:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698536.659057:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698537.683060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698538.707061:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698539.731077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698540.755022:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698541.203005:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698541.203017:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698541.203022:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698541.203036:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6442 00000100:00080000:2.0:1787698541.203050:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698541.203059:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698541.203109:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134612864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698541.203135:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698541.203138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698541.203139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698541.203142:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6442 00000100:00080000:2.0:1787698541.203145:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698541.203147:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698541.203157:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698541.203170:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134612992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698541.203180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698541.203183:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698541.203186:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698541.203190:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6442 00000100:00080000:2.0:1787698541.203193:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698541.203199:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698541.203201:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134612864:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787698541.203206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698541.203217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134613120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698541.203224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698541.203226:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698541.203228:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698541.203230:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6442 00000100:00080000:2.0:1787698541.203233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698541.203235:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698541.203238:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698541.203245:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134613248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698541.203251:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698541.203253:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698541.203254:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698541.203257:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6442 00000100:00080000:2.0:1787698541.203259:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698541.203262:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698541.203265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698541.203272:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134613376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698541.203277:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698541.203279:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698541.203280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698541.203283:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6442 00000100:00080000:2.0:1787698541.203285:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698541.203287:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698541.203290:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698541.203295:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134613504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698541.203303:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698541.203305:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698541.203309:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698541.203310:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698541.203313:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6442 00000100:00080000:2.0:1787698541.203315:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698541.203317:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698541.203319:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698541.203326:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134613632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698541.203357:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698541/real 1787698541] req@ffff9d85babd4380 x1874531134612864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698541.203379:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134612864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698541.203389:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698541.203393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698541.203397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134612864:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698541.203405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612992:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698541.203427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698541.203440:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698541/real 1787698541] req@ffff9d85babd4000 x1874531134612992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698541.203452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134612992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698541.203460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698541.203463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698541.203466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134612992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698541.203496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134613120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698541.203517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698541.203527:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698541/real 1787698541] req@ffff9d85babd5c00 x1874531134613120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698541.203538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134613120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698541.203552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698541.203555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698541.203558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134613120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698541.203563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134613248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698541.203585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698541.203595:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698541/real 1787698541] req@ffff9d85babd4700 x1874531134613248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698541.203606:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134613248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698541.203614:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698541.203618:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698541.203621:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134613248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698541.203626:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134613376:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698541.203646:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698541.203657:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698541/real 1787698541] req@ffff9d85babd5f80 x1874531134613376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698541.203668:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134613376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698541.203676:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698541.203679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698541.203683:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134613376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698541.203689:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134613504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698541.203706:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698541.203716:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698541/real 1787698541] req@ffff9d85babd7100 x1874531134613504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698541.203727:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134613504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698541.203768:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698541.203771:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698541.203775:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134613504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698541.203781:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134613632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698541.203800:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698541.203809:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698541/real 1787698541] req@ffff9d85babd4a80 x1874531134613632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698541.203820:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134613632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698541.203828:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698541.203831:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698541.203834:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134613632:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698541.779070:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698542.803060:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698543.827054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698544.851044:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698545.875018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698546.323015:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698546.323025:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698546.323030:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698546.323040:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6448 00000100:00080000:2.0:1787698546.323053:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698546.323062:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698546.323102:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134613760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698546.323122:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698546.323125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698546.323126:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698546.323130:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6448 00000100:00080000:2.0:1787698546.323133:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698546.323135:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698546.323144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698546.323152:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134613760:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698546.323155:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134613888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698546.323165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698546.323167:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698546.323168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698546.323171:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6448 00000100:00080000:2.0:1787698546.323172:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698546.323175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698546.323178:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698546.323184:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134614016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698546.323190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698546.323192:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698546.323193:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698546.323195:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6448 00000100:00080000:2.0:1787698546.323197:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698546.323199:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698546.323202:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698546.323209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134614144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698546.323215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698546.323217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698546.323218:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698546.323220:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6448 00000100:00080000:2.0:1787698546.323223:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698546.323226:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698546.323229:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698546.323233:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698546.323246:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134614272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698546.323251:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698546.323256:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698546.323257:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698546.323260:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6448 00000100:00080000:2.0:1787698546.323263:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698546.323266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698546.323269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698546.323278:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134614400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698546.323280:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698546/real 1787698546] req@ffff9d85babd5180 x1874531134613760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698546.323290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698546.323297:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698546.323299:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698546.323304:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6448 00000100:00080000:2.0:1787698546.323306:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698546.323309:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698546.323311:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698546.323316:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134613760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698546.323320:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134614528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698546.323341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698546.323348:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698546.323353:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134613760:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698546.323379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134613888:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698546.323403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698546.323416:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698546/real 1787698546] req@ffff9d85babd4e00 x1874531134613888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698546.323429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134613888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698546.323435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698546.323437:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698546.323439:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134613888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698546.323442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698546.323457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698546.323464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698546/real 1787698546] req@ffff9d85babd7b80 x1874531134614016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698546.323471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134614016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698546.323481:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698546.323484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698546.323487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698546.323492:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698546.323509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698546.323519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698546/real 1787698546] req@ffff9d85babd4a80 x1874531134614144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698546.323530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134614144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698546.323537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698546.323541:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698546.323544:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698546.323549:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134614272:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698546.323568:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698546.323579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698546/real 1787698546] req@ffff9d85babd7100 x1874531134614272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698546.323591:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134614272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698546.323599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698546.323602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698546.323606:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134614272:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698546.323611:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134614400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698546.323628:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698546.323637:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698546/real 1787698546] req@ffff9d85babd5f80 x1874531134614400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698546.323660:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134614400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698546.323668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698546.323671:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698546.323674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134614400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698546.323679:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134614528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698546.323695:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698546.323703:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698546/real 1787698546] req@ffff9d85babd4700 x1874531134614528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698546.323713:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134614528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698546.323718:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698546.323720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698546.323722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134614528:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698546.899080:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698547.923076:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698548.947053:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698549.971037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698550.995039:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698551.442978:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698551.442994:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698551.442999:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698551.443012:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6453 00000100:00080000:2.0:1787698551.443025:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698551.443034:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698551.443083:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134614656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698551.443112:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698551.443115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698551.443117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698551.443122:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6453 00000100:00080000:2.0:1787698551.443126:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698551.443131:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698551.443151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698551.443168:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134614656:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698551.443170:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134614784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698551.443191:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698551.443195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698551.443197:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698551.443202:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6453 00000100:00080000:2.0:1787698551.443206:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698551.443210:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698551.443214:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698551.443226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134614912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698551.443237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698551.443240:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698551.443243:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698551.443248:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6453 00000800:00000100:0.0:1787698551.443255:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698551.443256:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698551.443268:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698551.443275:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698551.443289:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134615040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698551.443300:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698551.443304:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698551.443306:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698551.443314:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698551/real 1787698551] req@ffff9d85babd5c00 x1874531134614656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698551.443314:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6453 00000100:00080000:2.0:1787698551.443328:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698551.443340:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698551.443349:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698551.443353:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134614656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698551.443365:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698551.443371:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134615168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698551.443374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698551.443382:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134614656:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698551.443408:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698551.443412:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698551.443415:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698551.443421:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6453 00000100:00080000:2.0:1787698551.443426:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698551.443431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614784:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698551.443436:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698551.443445:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698551.443456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698551.443463:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134615296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698551.443472:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698551/real 1787698551] req@ffff9d85babd4000 x1874531134614784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698551.443484:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698551.443497:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698551.443500:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698551.443510:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6453 00000100:00100000:0.0:1787698551.443512:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134614784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698551.443517:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698551.443526:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698551.443529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698551.443533:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698551.443538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698551.443545:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614784:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698551.443566:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134615424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698551.443567:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698551.443605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698551.443621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698551/real 1787698551] req@ffff9d85babd4380 x1874531134614912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698551.443634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134614912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698551.443648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698551.443651:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698551.443655:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134614912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698551.443661:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698551.443679:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698551.443690:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698551/real 1787698551] req@ffff9d85babd6300 x1874531134615040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698551.443701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134615040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698551.443709:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698551.443712:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698551.443716:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698551.443722:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134615168:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698551.443738:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698551.443748:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698551/real 1787698551] req@ffff9d85babd7480 x1874531134615168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698551.443760:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134615168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698551.443767:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698551.443770:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698551.443774:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134615168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698551.443791:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134615296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698551.443809:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698551.443818:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698551/real 1787698551] req@ffff9d85babd6680 x1874531134615296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698551.443828:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134615296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698551.443835:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698551.443838:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698551.443841:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134615296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698551.443847:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134615424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698551.443908:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698551.443918:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698551/real 1787698551] req@ffff9d85babd6a00 x1874531134615424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698551.443929:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134615424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698551.443943:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698551.443946:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698551.443950:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134615424:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698552.019092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698553.043026:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698554.067077:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698555.090964:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698556.115032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698556.562993:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698556.563003:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698556.563006:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698556.563017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6458 00000100:00080000:2.0:1787698556.563030:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698556.563038:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698556.563080:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134615552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698556.563109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698556.563112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698556.563117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698556.563121:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6458 00000100:00080000:2.0:1787698556.563123:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698556.563126:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698556.563136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698556.563145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134615680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698556.563151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698556.563154:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698556.563155:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698556.563157:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6458 00000100:00080000:2.0:1787698556.563159:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698556.563160:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698556.563164:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698556.563168:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134615552:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698556.563172:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134615808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698556.563181:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698556.563184:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698556.563185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698556.563187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6458 00000100:00080000:2.0:1787698556.563189:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698556.563191:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698556.563194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698556.563204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134615936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698556.563209:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698556.563211:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698556.563212:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698556.563214:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6458 00000100:00080000:2.0:1787698556.563217:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698556.563220:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698556.563223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698556.563229:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134616064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698556.563235:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698556.563237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698556.563238:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698556.563241:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6458 00000100:00080000:2.0:1787698556.563244:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698556.563245:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698556.563251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698556.563257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698556.563269:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134616192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698556.563285:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698556.563288:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698556.563289:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698556.563292:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698556/real 1787698556] req@ffff9d85babd5500 x1874531134615552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698556.563295:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6458 00000100:00080000:2.0:1787698556.563301:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698556.563307:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698556.563312:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698556.563315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134615552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698556.563324:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698556.563327:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134616320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698556.563331:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698556.563339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134615552:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698556.563348:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615680:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698556.563363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698556.563372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698556/real 1787698556] req@ffff9d85babd6d80 x1874531134615680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698556.563380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134615680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698556.563385:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698556.563388:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698556.563389:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698556.563414:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698556.563437:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698556.563445:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698556/real 1787698556] req@ffff9d85babd7800 x1874531134615808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698556.563452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134615808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698556.563461:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698556.563463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698556.563465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698556.563481:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698556.563494:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698556.563500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698556/real 1787698556] req@ffff9d85babd6a00 x1874531134615936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698556.563507:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134615936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698556.563512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698556.563514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698556.563516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134615936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698556.563519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616064:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698556.563529:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698556.563534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698556/real 1787698556] req@ffff9d85babd6680 x1874531134616064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698556.563541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134616064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698556.563545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698556.563547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698556.563549:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698556.563552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698556.563561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698556.563565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698556/real 1787698556] req@ffff9d85babd7480 x1874531134616192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698556.563571:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134616192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698556.563575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698556.563576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698556.563578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698556.563581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698556.563589:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698556.563594:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698556/real 1787698556] req@ffff9d85babd6300 x1874531134616320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698556.563599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134616320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698556.563604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698556.563606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698556.563607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616320:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698557.139123:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698558.163000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698559.187065:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698560.210995:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698561.235087:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698561.682984:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698561.682994:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698561.682997:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698561.683007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6463 00000100:00080000:2.0:1787698561.683019:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698561.683026:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698561.683067:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134616448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698561.683098:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698561.683100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698561.683102:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698561.683106:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6463 00000100:00080000:2.0:1787698561.683109:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698561.683111:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698561.683120:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698561.683130:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134616576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698561.683137:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698561.683139:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698561.683140:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698561.683142:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6463 00000100:00080000:2.0:1787698561.683143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698561.683145:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698561.683148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698561.683154:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134616704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698561.683160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698561.683163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698561.683164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698561.683166:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6463 00000100:00080000:2.0:1787698561.683168:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698561.683170:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698561.683172:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698561.683176:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134616448:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698561.683182:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134616832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698561.683193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698561.683196:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698561.683198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698561.683202:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6463 00000100:00080000:2.0:1787698561.683206:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698561.683210:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698561.683215:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698561.683227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134616960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698561.683235:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698561.683239:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698561.683241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698561.683245:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6463 00000100:00080000:2.0:1787698561.683249:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698561.683254:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698561.683259:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698561.683270:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134617088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698561.683274:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698561.683284:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698561.683286:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698561.683288:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698561.683291:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6463 00000100:00080000:2.0:1787698561.683295:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698561.683298:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698561.683302:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698561.683310:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134617216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698561.683334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698561/real 1787698561] req@ffff9d85babd4380 x1874531134616448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698561.683355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134616448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698561.683366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698561.683370:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698561.683374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134616448:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698561.683394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134616576:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698561.683418:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698561.683431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698561/real 1787698561] req@ffff9d85babd4000 x1874531134616576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698561.683443:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134616576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698561.683451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698561.683454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698561.683458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134616576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698561.683464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134616704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698561.683484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698561.683495:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698561/real 1787698561] req@ffff9d85babd5c00 x1874531134616704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698561.683506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134616704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698561.683520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698561.683523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698561.683526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134616704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698561.683552:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134616832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698561.683573:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698561.683584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698561/real 1787698561] req@ffff9d85babd4700 x1874531134616832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698561.683595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134616832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698561.683603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698561.683606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698561.683609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134616832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698561.683614:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616960:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698561.683633:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698561.683644:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698561/real 1787698561] req@ffff9d85babd5f80 x1874531134616960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698561.683655:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134616960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698561.683662:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698561.683664:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698561.683668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134616960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698561.683673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698561.683690:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698561.683699:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698561/real 1787698561] req@ffff9d85babd7100 x1874531134617088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698561.683709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134617088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698561.683717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698561.683720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698561.683722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698561.683727:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698561.683743:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698561.683752:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698561/real 1787698561] req@ffff9d85babd4a80 x1874531134617216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698561.683762:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134617216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698561.683770:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698561.683773:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698561.683776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617216:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698562.259009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698563.283027:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698564.307094:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698565.331028:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698566.355043:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698566.802993:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698566.803008:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698566.803013:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698566.803027:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6468 00000100:00080000:2.0:1787698566.803043:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698566.803053:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698566.803112:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134617344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698566.803152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698566.803157:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698566.803159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698566.803165:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6468 00000100:00080000:2.0:1787698566.803169:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698566.803175:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698566.803188:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698566.803206:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134617344:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698566.803210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134617472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698566.803230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698566.803233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698566.803235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698566.803240:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6468 00000100:00080000:2.0:1787698566.803244:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698566.803249:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698566.803256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698566.803273:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134617600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698566.803284:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698566.803288:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698566.803290:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698566.803312:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6468 00000100:00080000:2.0:1787698566.803318:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698566.803322:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698566.803329:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698566.803334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698566.803371:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134617728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698566.803384:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698566.803388:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698566.803390:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698566.803392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698566/real 1787698566] req@ffff9d85babd7b80 x1874531134617344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698566.803398:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6468 00000100:00080000:2.0:1787698566.803408:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698566.803415:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698566.803421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134617344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698566.803430:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698566.803434:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698566.803440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698566.803445:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134617344:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698566.803451:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134617856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698566.803464:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698566.803467:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698566.803470:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698566.803474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134617472:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698566.803477:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6468 00000100:00080000:2.0:1787698566.803485:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698566.803491:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698566.803497:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698566.803498:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698566.803507:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698566/real 1787698566] req@ffff9d85babd4e00 x1874531134617472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698566.803519:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134617984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698566.803521:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134617472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698566.803552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698566.803557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698566.803558:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698566.803563:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698566.803565:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698566.803571:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134617472:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698566.803574:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6468 00000100:00080000:2.0:1787698566.803582:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698566.803583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134617600:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698566.803591:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698566.803598:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698566.803610:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698566.803617:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134618112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698566.803625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698566/real 1787698566] req@ffff9d85babd5180 x1874531134617600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698566.803659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134617600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698566.803674:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698566.803678:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698566.803681:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134617600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698566.803688:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134617728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698566.803709:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698566.803720:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698566/real 1787698566] req@ffff9d85babd4a80 x1874531134617728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698566.803739:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134617728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698566.803747:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698566.803750:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698566.803753:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134617728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698566.803759:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617856:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698566.803778:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698566.803788:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698566/real 1787698566] req@ffff9d85babd7100 x1874531134617856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698566.803798:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134617856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698566.803814:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698566.803818:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698566.803822:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698566.803840:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698566.803891:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698566.803902:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698566/real 1787698566] req@ffff9d85babd5f80 x1874531134617984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698566.803913:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134617984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698566.803921:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698566.803924:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698566.803927:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134617984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698566.803933:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134618112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698566.803949:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698566.803958:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698566/real 1787698566] req@ffff9d85babd4700 x1874531134618112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698566.803969:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134618112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698566.803977:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698566.803980:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698566.803983:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134618112:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698567.379066:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698568.402998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698569.427088:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698570.450992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698571.475039:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698571.923006:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698571.923016:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698571.923021:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698571.923032:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6473 00000100:00080000:0.0:1787698571.923046:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698571.923054:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698571.923108:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134618240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698571.923149:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698571.923152:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698571.923153:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698571.923157:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6473 00000100:00080000:0.0:1787698571.923160:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698571.923163:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698571.923173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698571.923184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134618368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698571.923192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698571.923194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698571.923196:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698571.923198:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6473 00000100:00080000:0.0:1787698571.923200:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698571.923202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698571.923216:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698571.923226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134618496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698571.923238:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134618240:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698571.923242:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698571.923254:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698571.923256:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698571.923261:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6473 00000100:00080000:0.0:1787698571.923272:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698571.923278:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698571.923284:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698571.923300:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134618624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698571.923310:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698571.923311:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698571.923322:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698571.923323:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698571.923329:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6473 00000100:00080000:0.0:1787698571.923344:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698571.923350:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698571.923357:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698571.923360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698571/real 1787698571] req@ffff9d85b9d08e00 x1874531134618240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698571.923376:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134618752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698571.923384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134618240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698571.923390:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698571.923398:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698571.923403:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698571.923410:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6473 00000100:00080000:0.0:1787698571.923416:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698571.923422:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698571.923429:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698571.923443:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134618880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698571.923445:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698571.923457:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698571.923469:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698571.923470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134618240:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698571.923474:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698571.923477:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698571.923484:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6473 00000100:00080000:0.0:1787698571.923487:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698571.923491:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698571.923495:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698571.923497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134618368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698571.923508:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134619008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698571.923534:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698571.923558:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698571/real 1787698571] req@ffff9d85b9d08a80 x1874531134618368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698571.923572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134618368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698571.923584:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698571.923588:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698571.923591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134618368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698571.923617:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134618496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698571.923638:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698571.923649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698571/real 1787698571] req@ffff9d85b9d08000 x1874531134618496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698571.923659:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134618496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698571.923667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698571.923670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698571.923673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134618496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698571.923679:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134618624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698571.923696:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698571.923706:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698571/real 1787698571] req@ffff9d85b9d0ad80 x1874531134618624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698571.923716:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134618624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698571.923724:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698571.923727:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698571.923729:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134618624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698571.923735:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134618752:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698571.923752:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698571.923762:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698571/real 1787698571] req@ffff9d85b9d0bb80 x1874531134618752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698571.923771:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134618752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698571.923788:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698571.923791:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698571.923794:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134618752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698571.923801:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134618880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698571.923821:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698571.923833:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698571/real 1787698571] req@ffff9d85b9d0a680 x1874531134618880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698571.923843:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134618880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698571.923887:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698571.923891:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698571.923894:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134618880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698571.923900:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698571.923919:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698571.923929:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698571/real 1787698571] req@ffff9d85b9d0b480 x1874531134619008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698571.923948:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134619008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698571.923956:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698571.923959:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698571.923962:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619008:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698572.499026:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698573.523015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698574.547079:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698575.571004:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698576.595075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698577.042977:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698577.042987:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698577.042992:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698577.043004:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6478 00000100:00080000:0.0:1787698577.043017:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698577.043025:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698577.043072:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134619136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698577.043101:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698577.043109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698577.043110:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698577.043113:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6478 00000100:00080000:0.0:1787698577.043116:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698577.043129:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698577.043139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698577.043150:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134619264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698577.043158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698577.043161:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134619136:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698577.043162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698577.043164:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698577.043171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6478 00000100:00080000:0.0:1787698577.043174:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698577.043176:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698577.043179:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698577.043187:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134619392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698577.043192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698577.043195:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698577.043197:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698577.043199:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6478 00000100:00080000:0.0:1787698577.043201:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698577.043203:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698577.043208:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698577.043215:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134619520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698577.043220:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698577.043223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698577.043224:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698577.043226:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6478 00000100:00080000:0.0:1787698577.043228:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698577.043230:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698577.043233:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698577.043239:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134619648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698577.043244:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698577.043247:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698577.043248:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698577.043250:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6478 00000100:00080000:0.0:1787698577.043252:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698577.043254:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698577.043256:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698577.043263:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134619776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698577.043268:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698577.043270:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698577.043272:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698577.043275:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6478 00000100:00080000:0.0:1787698577.043277:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698577.043279:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698577.043282:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698577.043287:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134619264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698577.043291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134619904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698577.043323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134619392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698577.043346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134619520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698577.043363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698577.043394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698577.043419:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619904:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698577.619011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698578.643046:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698579.667032:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787698580.115257:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:0.0:1787698580.115323:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787698580.115346:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787698580.115411:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787698580.115426:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787698580.115436:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787698580.115446:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787698580.115455:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:0.0:1787698580.115464:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698580.115778:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698577/real 1787698580] req@ffff9d85b9d09c00 x1874531134619904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698580.115816:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134619904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698580.115834:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698580.115842:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698580.115848:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698580.115886:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698577/real 1787698580] req@ffff9d85b9d08700 x1874531134619264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698580.115902:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134619264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698580.115912:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698580.115918:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698580.115921:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134619264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698580.115929:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698577/real 1787698580] req@ffff9d85b9d09500 x1874531134619392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698580.115940:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134619392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698580.115954:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698580.115958:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698580.115961:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134619392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698580.115970:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698577/real 1787698580] req@ffff9d85b9d09180 x1874531134619520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698580.115980:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134619520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698580.115989:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698580.115992:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698580.115995:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134619520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698580.116003:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698577/real 1787698580] req@ffff9d85b9d0a300 x1874531134619648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698580.116016:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134619648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698580.116023:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698580.116028:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698580.116031:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698580.116041:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698577/real 1787698580] req@ffff9d85b9d09f80 x1874531134619776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698580.116052:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134619776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698580.116061:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698580.116080:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698580.116083:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134619776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698580.116091:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698577/real 1787698580] req@ffff9d85b9d0b800 x1874531134619136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698580.116107:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134619136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698580.116114:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698580.116119:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698580.116124:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134619136:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787698580.691046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698581.715090:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698582.162954:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698582.162967:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698582.162970:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698582.162981:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6483 00000100:00080000:0.0:1787698582.162992:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698582.163000:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698582.163063:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134620032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698582.163095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698582.163100:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698582.163102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698582.163106:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6483 00000100:00080000:0.0:1787698582.163110:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698582.163115:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698582.163126:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698582.163142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134620160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698582.163154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698582.163154:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134620032:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698582.163162:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698582.163166:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698582.163171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6483 00000100:00080000:0.0:1787698582.163175:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698582.163180:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698582.163186:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698582.163200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134620288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698582.163211:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698582.163215:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698582.163217:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698582.163220:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6483 00000100:00080000:0.0:1787698582.163234:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698582.163239:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698582.163245:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698582.163260:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134620416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698582.163262:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698582.163274:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698582.163278:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698582.163280:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698582.163284:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6483 00000100:00080000:0.0:1787698582.163296:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698582.163302:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698582.163308:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698582.163321:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134620544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698582.163333:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698582.163333:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698582/real 1787698582] req@ffff9d85b9d0b800 x1874531134620032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698582.163343:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698582.163348:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698582.163357:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6483 00000100:00080000:0.0:1787698582.163362:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698582.163369:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698582.163369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134620032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698582.163378:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698582.163387:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698582.163394:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134620672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698582.163395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698582.163404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134620032:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698582.163411:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698582.163416:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698582.163418:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698582.163423:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6483 00000100:00080000:0.0:1787698582.163427:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698582.163432:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698582.163437:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698582.163440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134620160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698582.163451:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134620800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698582.163473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698582.163488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698582/real 1787698582] req@ffff9d85b9d09f80 x1874531134620160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698582.163501:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134620160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698582.163508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698582.163511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698582.163514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134620160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698582.163520:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134620288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698582.163540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698582.163551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698582/real 1787698582] req@ffff9d85b9d0a300 x1874531134620288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698582.163562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134620288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698582.163570:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698582.163574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698582.163577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134620288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698582.163582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134620416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698582.163598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698582.163607:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698582/real 1787698582] req@ffff9d85b9d09180 x1874531134620416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698582.163626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134620416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698582.163634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698582.163636:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698582.163639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134620416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698582.163643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134620544:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698582.163659:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698582.163670:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698582/real 1787698582] req@ffff9d85b9d09500 x1874531134620544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698582.163681:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134620544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698582.163689:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698582.163692:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698582.163695:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134620544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698582.163700:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134620672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698582.163717:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698582.163728:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698582/real 1787698582] req@ffff9d85b9d08700 x1874531134620672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698582.163739:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134620672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698582.163753:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698582.163757:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698582.163760:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134620672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698582.163779:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134620800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698582.163800:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698582.163813:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698582/real 1787698582] req@ffff9d85b9d09c00 x1874531134620800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698582.163825:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134620800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698582.163833:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698582.163836:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698582.163840:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134620800:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698582.739061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698583.762988:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698584.787067:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698585.811068:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698586.835053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698587.283004:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698587.283017:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698587.283022:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698587.283034:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6489 00000100:00080000:1.0:1787698587.283047:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698587.283055:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698587.283099:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134620928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698587.283130:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698587.283135:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698587.283137:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698587.283143:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6489 00000100:00080000:1.0:1787698587.283147:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698587.283152:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283163:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134620928:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787698587.283164:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698587.283183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134621056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698587.283195:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698587.283198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698587.283200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698587.283204:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6489 00000100:00080000:1.0:1787698587.283208:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698587.283212:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698587.283218:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698587.283229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134621184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698587.283237:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698587.283242:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698587.283245:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698587.283248:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698587.283254:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6489 00000100:00080000:1.0:1787698587.283259:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698587.283263:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698587.283269:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698587.283282:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134621312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698587.283288:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698587/real 1787698587] req@ffff9d85b9d2dc00 x1874531134620928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698587.283296:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698587.283301:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698587.283303:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698587.283307:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134620928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698587.283308:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6489 00000100:00080000:1.0:1787698587.283315:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698587.283320:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698587.283323:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698587.283326:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698587.283331:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134620928:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787698587.283331:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698587.283347:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134621440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698587.283357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621056:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698587.283360:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698587.283365:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698587.283367:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698587.283372:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6489 00000800:00000100:0.0:1787698587.283376:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698587.283380:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698587.283387:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698587/real 1787698587] req@ffff9d85b9d2c380 x1874531134621056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698587.283395:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698587.283410:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134621568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698587.283414:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134621056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698587.283431:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698587.283439:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698587.283443:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698587.283450:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6489 00000100:00080000:0.0:1787698587.283452:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698587.283454:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698587.283459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698587.283462:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621056:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787698587.283470:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698587.283474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698587.283489:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134621696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698587.283498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698587/real 1787698587] req@ffff9d85b9d2c700 x1874531134621184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698587.283517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134621184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698587.283523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698587.283525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698587.283527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698587.283532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698587.283543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698587/real 1787698587] req@ffff9d85b9d2ed80 x1874531134621312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698587.283554:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134621312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698587.283559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698587.283560:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698587.283563:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698587.283573:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134621440:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698587.283583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698587/real 1787698587] req@ffff9d85b9d2d500 x1874531134621440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698587.283595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134621440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698587.283606:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698587.283608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698587.283610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134621440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698587.283613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134621568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698587.283623:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283629:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698587/real 1787698587] req@ffff9d85b9d2e680 x1874531134621568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698587.283635:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134621568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698587.283640:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698587.283641:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698587.283643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134621568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698587.283646:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134621696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698587.283654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698587.283659:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698587/real 1787698587] req@ffff9d85b9d2dc00 x1874531134621696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698587.283666:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134621696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698587.283671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698587.283672:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698587.283674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134621696:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698587.859030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698588.883078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698589.907016:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698590.931036:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698591.955053:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698592.403003:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698592.403012:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698592.403017:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698592.403027:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6494 00000100:00080000:1.0:1787698592.403038:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698592.403046:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698592.403086:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134621824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698592.403112:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698592.403115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698592.403116:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698592.403119:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6494 00000100:00080000:1.0:1787698592.403121:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698592.403123:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698592.403131:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698592.403142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134621952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698592.403154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698592.403157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698592.403159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698592.403162:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134621824:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698592.403164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6494 00000100:00080000:1.0:1787698592.403169:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698592.403172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698592.403175:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698592.403182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134622080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698592.403188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698592.403190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698592.403191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698592.403193:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6494 00000100:00080000:1.0:1787698592.403195:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698592.403197:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698592.403200:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698592.403211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134622208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698592.403219:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698592.403222:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698592.403224:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698592.403228:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6494 00000800:00000100:0.0:1787698592.403238:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698592.403241:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698592.403248:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698592.403254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698592.403265:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134622336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698592.403272:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698592.403274:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698592.403276:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698592.403278:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6494 00000100:00080000:1.0:1787698592.403281:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698592.403284:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698592.403287:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698592.403294:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698592/real 1787698592] req@ffff9d85b9d2e300 x1874531134621824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698592.403296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134622464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698592.403314:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698592.403318:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698592.403319:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698592.403322:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6494 00000100:00080000:1.0:1787698592.403324:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698592.403329:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134621824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698592.403338:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698592.403341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787698592.403344:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698592.403346:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698592.403352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134621824:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698592.403362:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134622592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698592.403375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698592.403411:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698592.403425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698592/real 1787698592] req@ffff9d85b9d2ce00 x1874531134621952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698592.403437:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134621952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698592.403458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698592.403461:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698592.403465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134621952:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698592.403471:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698592.403491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698592.403503:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698592/real 1787698592] req@ffff9d85b9d2c000 x1874531134622080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698592.403514:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134622080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698592.403522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698592.403525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698592.403528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698592.403532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698592.403546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698592.403552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698592/real 1787698592] req@ffff9d85b9d2ca80 x1874531134622208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698592.403559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134622208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698592.403564:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698592.403566:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698592.403568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698592.403572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134622336:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698592.403581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698592.403586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698592/real 1787698592] req@ffff9d85b9d2f800 x1874531134622336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698592.403592:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134622336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698592.403597:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698592.403598:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698592.403600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134622336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698592.403603:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134622464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698592.403611:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698592.403615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698592/real 1787698592] req@ffff9d85b9d2fb80 x1874531134622464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698592.403621:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134622464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698592.403626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698592.403628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698592.403629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134622464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698592.403641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134622592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698592.403651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698592.403656:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698592/real 1787698592] req@ffff9d85b9d2f480 x1874531134622592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698592.403663:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134622592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698592.403668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698592.403669:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698592.403671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134622592:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698592.979030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698594.003042:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698595.027005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698596.051078:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698597.075007:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698597.522984:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698597.522992:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698597.522995:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698597.523003:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6499 00000100:00080000:0.0:1787698597.523013:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698597.523021:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698597.523061:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134622720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698597.523100:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698597.523104:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698597.523107:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698597.523112:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6499 00000100:00080000:0.0:1787698597.523117:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698597.523120:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134622720:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698597.523125:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698597.523139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698597.523157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134622848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698597.523170:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698597.523174:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698597.523176:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698597.523181:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6499 00000800:00000100:1.0:1787698597.523187:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698597.523189:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698597.523196:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698597.523204:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698597.523219:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134622976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698597.523230:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698597/real 1787698597] req@ffff9d85b9d0b480 x1874531134622720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698597.523231:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698597.523242:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698597.523246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698597.523254:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6499 00000100:00080000:0.0:1787698597.523258:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698597.523263:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134622720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698597.523265:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698597.523274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698597.523285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698597.523292:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134623104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698597.523293:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698597.523298:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134622720:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698597.523315:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698597.523319:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698597.523321:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698597.523325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622848:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698597.523331:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6499 00000100:00080000:0.0:1787698597.523338:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698597.523344:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698597.523350:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698597.523355:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698597.523370:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134623232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698597.523375:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698597/real 1787698597] req@ffff9d85b9d0a680 x1874531134622848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698597.523399:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698597.523403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134622848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698597.523405:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698597.523409:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698597.523417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698597.523419:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6499 00000100:00080000:0.0:1787698597.523425:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698597.523429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698597.523432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622848:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698597.523442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622976:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698597.523443:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698597.523453:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698597.523465:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698597.523473:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134623360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698597.523477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698597/real 1787698597] req@ffff9d85b9d0bb80 x1874531134622976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698597.523500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134622976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698597.523510:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698597.523513:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698597.523513:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698597.523520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134622976:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698597.523524:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698597.523528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134623104:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698597.523529:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698597.523538:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6499 00000100:00080000:0.0:1787698597.523544:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698597.523545:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698597.523554:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698597.523557:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698597/real 1787698597] req@ffff9d85b9d0ad80 x1874531134623104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698597.523564:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698597.523567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134623104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698597.523575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698597.523577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698597.523580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134623104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698597.523586:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134623488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698597.523595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134623232:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698597.523611:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698597.523622:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698597/real 1787698597] req@ffff9d85b9d08000 x1874531134623232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698597.523632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134623232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698597.523638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698597.523640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698597.523642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134623232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698597.523645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134623360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698597.523654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698597.523660:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698597/real 1787698597] req@ffff9d85b9d08a80 x1874531134623360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698597.523667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134623360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698597.523672:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698597.523674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698597.523675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134623360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698597.523683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134623488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698597.523692:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698597.523698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698597/real 1787698597] req@ffff9d85b9d08e00 x1874531134623488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698597.523705:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134623488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698597.523710:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698597.523711:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698597.523714:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134623488:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698598.099068:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698599.123031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698600.147049:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698601.170983:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698602.195040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698602.642959:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698602.642970:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698602.642974:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698602.642985:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6504 00000100:00080000:0.0:1787698602.642996:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698602.643004:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698602.643043:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134623616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698602.643080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698602.643084:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698602.643085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698602.643089:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6504 00000100:00080000:0.0:1787698602.643092:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698602.643095:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698602.643105:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698602.643123:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134623616:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698602.643130:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134623744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698602.643144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698602.643148:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698602.643151:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698602.643155:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6504 00000100:00080000:0.0:1787698602.643159:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698602.643162:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698602.643168:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698602.643178:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134623872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698602.643185:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698602.643190:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698602.643191:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698602.643194:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698602.643198:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6504 00000100:00080000:0.0:1787698602.643202:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698602.643204:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698602.643208:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698602.643217:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134624000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698602.643228:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698602/real 1787698602] req@ffff9d85b9d08e00 x1874531134623616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698602.643230:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698602.643239:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698602.643243:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698602.643249:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6504 00000100:00080000:0.0:1787698602.643252:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698602.643257:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698602.643260:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698602.643261:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134623616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698602.643269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134624128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698602.643273:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698602.643281:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698602.643282:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698602.643288:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698602.643289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134623616:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698602.643292:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698602.643297:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6504 00000100:00080000:0.0:1787698602.643299:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698602.643302:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698602.643305:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698602.643312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134624256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698602.643316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134623744:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698602.643325:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698602.643328:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698602.643329:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698602.643334:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6504 00000100:00080000:0.0:1787698602.643337:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698602.643341:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698602.643341:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698602.643348:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698602.643356:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698602/real 1787698602] req@ffff9d85b9d08a80 x1874531134623744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698602.643360:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134624384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698602.643380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134623744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698602.643394:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698602.643397:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698602.643399:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134623744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698602.643405:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134623872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698602.643420:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698602.643427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698602/real 1787698602] req@ffff9d85b9d08000 x1874531134623872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698602.643435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134623872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698602.643440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698602.643441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698602.643443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134623872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698602.643446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698602.643456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698602.643462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698602/real 1787698602] req@ffff9d85b9d0ad80 x1874531134624000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698602.643469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134624000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698602.643474:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698602.643475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698602.643477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698602.643480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134624128:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698602.643489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698602.643494:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698602/real 1787698602] req@ffff9d85b9d0bb80 x1874531134624128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698602.643502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134624128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698602.643507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698602.643515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698602.643517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134624128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698602.643528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134624256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698602.643539:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698602.643545:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698602/real 1787698602] req@ffff9d85b9d0a680 x1874531134624256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698602.643552:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134624256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698602.643557:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698602.643559:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698602.643561:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134624256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698602.643564:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134624384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698602.643573:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698602.643579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698602/real 1787698602] req@ffff9d85b9d0b480 x1874531134624384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698602.643586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134624384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698602.643591:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698602.643592:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698602.643595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134624384:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698603.219036:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698604.243079:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698605.266984:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698606.291097:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698607.315048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698607.763024:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698607.763038:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698607.763044:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698607.763057:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6509 00000100:00080000:0.0:1787698607.763072:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698607.763081:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698607.763139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134624512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698607.763176:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698607.763179:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698607.763180:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698607.763184:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6509 00000100:00080000:0.0:1787698607.763188:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698607.763192:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698607.763203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698607.763218:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134624640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698607.763229:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698607.763234:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698607.763236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134624512:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698607.763237:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698607.763245:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6509 00000100:00080000:0.0:1787698607.763249:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698607.763252:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698607.763256:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698607.763265:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134624768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698607.763271:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698607.763274:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698607.763275:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698607.763277:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6509 00000100:00080000:0.0:1787698607.763280:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698607.763283:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698607.763286:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698607.763292:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134624896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698607.763297:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698607.763300:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698607.763301:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698607.763303:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6509 00000100:00080000:0.0:1787698607.763315:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698607.763318:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698607.763320:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698607.763328:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134625024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698607.763333:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698607.763338:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698607.763342:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698607.763343:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698607.763346:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6509 00000100:00080000:0.0:1787698607.763348:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698607.763350:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698607.763353:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698607.763359:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134625152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698607.763369:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698607.763371:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698607.763372:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698607.763374:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6509 00000100:00080000:0.0:1787698607.763376:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698607.763379:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698607.763382:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698607.763389:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134625280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698607.763395:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698607/real 1787698607] req@ffff9d85b9d09c00 x1874531134624512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698607.763417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134624512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698607.763429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698607.763433:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698607.763437:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134624512:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698607.763463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624640:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698607.763488:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698607.763500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698607/real 1787698607] req@ffff9d85b9d08700 x1874531134624640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698607.763511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134624640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698607.763519:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698607.763523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698607.763526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698607.763532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698607.763553:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698607.763565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698607/real 1787698607] req@ffff9d85b9d09500 x1874531134624768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698607.763576:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134624768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698607.763584:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698607.763587:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698607.763590:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698607.763610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698607.763631:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698607.763643:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698607/real 1787698607] req@ffff9d85b9d09180 x1874531134624896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698607.763654:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134624896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698607.763670:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698607.763674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698607.763678:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134624896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698607.763683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625024:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698607.763703:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698607.763722:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698607/real 1787698607] req@ffff9d85b9d0a300 x1874531134625024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698607.763733:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134625024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698607.763741:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698607.763744:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698607.763747:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698607.763753:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698607.763771:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698607.763782:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698607/real 1787698607] req@ffff9d85b9d09f80 x1874531134625152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698607.763793:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134625152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698607.763800:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698607.763803:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698607.763806:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698607.763812:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698607.763830:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698607.763843:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698607/real 1787698607] req@ffff9d85b9d0b800 x1874531134625280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698607.763897:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134625280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698607.763906:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698607.763909:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698607.763913:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625280:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698608.339079:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698609.363006:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698610.387052:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698611.411053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698612.435087:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698612.882987:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698612.883001:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698612.883006:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698612.883020:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6514 00000100:00080000:2.0:1787698612.883034:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698612.883040:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698612.883089:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134625408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698612.883117:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698612.883120:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698612.883122:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698612.883127:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6514 00000100:00080000:2.0:1787698612.883130:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698612.883135:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698612.883146:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698612.883158:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134625536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698612.883167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698612.883170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698612.883171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698612.883175:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6514 00000100:00080000:2.0:1787698612.883179:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698612.883182:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698612.883187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698612.883207:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134625664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698612.883215:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698612.883218:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698612.883220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698612.883223:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6514 00000100:00080000:2.0:1787698612.883226:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698612.883229:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698612.883233:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698612.883242:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134625792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698612.883249:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698612.883252:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698612.883253:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698612.883256:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6514 00000100:00080000:2.0:1787698612.883259:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698612.883262:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698612.883266:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698612.883274:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134625920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698612.883286:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698612.883289:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698612.883291:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698612.883294:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6514 00000100:00080000:2.0:1787698612.883296:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698612.883300:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698612.883313:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698612.883323:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134626048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698612.883331:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698612.883334:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698612.883335:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698612.883338:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6514 00000100:00080000:2.0:1787698612.883341:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698612.883345:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698612.883349:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698612.883359:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134626176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698612.883605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134625408:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698612.883667:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698612.883719:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698612/real 1787698612] req@ffff9d85babd5c00 x1874531134625408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698612.883733:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134625408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698612.883741:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698612.883744:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698612.883748:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134625408:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698612.883752:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134625536:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698612.883767:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698612.883774:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698612/real 1787698612] req@ffff9d85babd4000 x1874531134625536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698612.883781:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134625536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698612.883786:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698612.883788:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698612.883790:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134625536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698612.883793:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134625664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698612.883803:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698612.883809:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698612/real 1787698612] req@ffff9d85babd4380 x1874531134625664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698612.883816:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134625664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698612.883820:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698612.883822:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698612.883823:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134625664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698612.883826:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134625792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698612.883835:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698612.883839:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698612/real 1787698612] req@ffff9d85babd6300 x1874531134625792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698612.883845:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134625792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698612.883882:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698612.883884:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698612.883886:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134625792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698612.883890:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625920:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698612.883903:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698612.883909:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698612/real 1787698612] req@ffff9d85babd7480 x1874531134625920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698612.883916:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134625920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698612.883921:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698612.883923:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698612.883925:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134625920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698612.883928:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698612.883937:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698612.883944:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698612/real 1787698612] req@ffff9d85babd6680 x1874531134626048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698612.883954:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134626048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698612.883961:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698612.883964:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698612.883968:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698612.883973:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698612.883991:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698612.884000:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698612/real 1787698612] req@ffff9d85babd6a00 x1874531134626176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698612.884007:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134626176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698612.884012:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698612.884014:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698612.884016:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626176:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698613.459082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698614.482978:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698615.507077:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698616.531505:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698617.555055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698618.003009:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698618.003019:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698618.003023:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698618.003033:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6519 00000100:00080000:3.0:1787698618.003045:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698618.003052:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698618.003093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134626304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698618.003117:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698618.003120:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698618.003121:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698618.003124:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6519 00000100:00080000:3.0:1787698618.003126:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698618.003128:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698618.003136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698618.003144:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134626432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698618.003149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698618.003152:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698618.003153:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698618.003155:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6519 00000100:00080000:3.0:1787698618.003156:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698618.003158:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698618.003161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698618.003166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134626560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698618.003171:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698618.003174:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698618.003176:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698618.003180:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6519 00000100:00080000:3.0:1787698618.003183:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698618.003187:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698618.003192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698618.003202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134626688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698618.003212:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698618.003214:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698618.003216:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698618.003220:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6519 00000100:00080000:3.0:1787698618.003223:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698618.003227:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698618.003231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698618.003242:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134626816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698618.003250:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698618.003254:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698618.003256:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698618.003259:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6519 00000100:00080000:3.0:1787698618.003262:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698618.003266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698618.003271:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698618.003292:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134626944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698618.003301:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698618.003304:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698618.003306:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698618.003309:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6519 00000100:00080000:3.0:1787698618.003312:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698618.003316:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698618.003327:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698618.003337:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134627072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698618.004501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134626304:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698618.004581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698618.004649:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698618/real 1787698618] req@ffff9d85c0fb3b80 x1874531134626304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698618.004671:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134626304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698618.004683:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698618.004688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698618.004693:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134626304:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698618.004701:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134626432:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698618.004724:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698618.004738:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698618/real 1787698618] req@ffff9d85c0fb0a80 x1874531134626432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698618.004750:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134626432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698618.004758:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698618.004761:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698618.004765:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134626432:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698618.004771:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134626560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698618.004791:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698618.004803:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698618/real 1787698618] req@ffff9d85c0fb1c00 x1874531134626560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698618.004815:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134626560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698618.004822:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698618.004825:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698618.004829:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134626560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698618.004835:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134626688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698618.004899:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698618.004912:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698618/real 1787698618] req@ffff9d85c0fb1f80 x1874531134626688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698618.004924:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134626688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698618.004932:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698618.004935:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698618.004939:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134626688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698618.004944:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626816:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698618.004964:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698618.004976:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698618/real 1787698618] req@ffff9d85c0fb3100 x1874531134626816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698618.004987:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134626816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698618.004996:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698618.004999:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698618.005003:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698618.005008:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698618.005025:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698618.005035:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698618/real 1787698618] req@ffff9d85c0fb1880 x1874531134626944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698618.005046:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134626944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698618.005054:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698618.005057:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698618.005060:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134626944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698618.005066:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698618.005081:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698618.005092:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698618/real 1787698618] req@ffff9d85c0fb1500 x1874531134627072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698618.005102:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134627072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698618.005110:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698618.005113:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698618.005116:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627072:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698618.579071:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698619.603194:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698620.627120:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698621.651070:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698622.675061:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698623.122936:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698623.122944:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698623.122947:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698623.122953:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6524 00000100:00080000:3.0:1787698623.122961:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698623.122966:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698623.122997:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134627200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698623.123023:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698623.123027:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698623.123029:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698623.123033:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6524 00000100:00080000:3.0:1787698623.123037:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698623.123040:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698623.123047:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698623.123058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134627328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698623.123065:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698623.123068:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698623.123070:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698623.123072:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134627200:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698623.123075:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6524 00000100:00080000:3.0:1787698623.123081:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698623.123082:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698623.123086:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698623.123095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134627456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698623.123101:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698623.123104:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698623.123105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698623.123107:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6524 00000100:00080000:3.0:1787698623.123109:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698623.123111:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698623.123114:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698623.123121:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134627584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698623.123131:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698623.123137:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698623.123140:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698623.123141:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698623.123145:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6524 00000100:00080000:3.0:1787698623.123147:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698623.123150:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698623.123153:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698623.123160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134627712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698623.123165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698623.123168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698623.123170:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698623.123172:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698623/real 1787698623] req@ffff9d85c0fb0700 x1874531134627200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698623.123174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6524 00000100:00080000:3.0:1787698623.123178:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698623.123181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698623.123188:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698623.123204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134627840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698623.123206:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134627200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698623.123227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698623.123230:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698623.123230:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698623.123232:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698623.123239:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698623.123240:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6524 00000100:00080000:3.0:1787698623.123247:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698623.123249:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134627200:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698623.123252:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698623.123260:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698623.123269:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134627968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698623.123279:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134627328:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698623.123297:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698623.123306:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698623/real 1787698623] req@ffff9d85c0fb3480 x1874531134627328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698623.123313:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134627328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698623.123319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698623.123321:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698623.123323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134627328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698623.123326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134627456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698623.123352:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698623.123359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698623/real 1787698623] req@ffff9d85c0fb2300 x1874531134627456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698623.123366:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134627456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698623.123371:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698623.123373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698623.123375:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134627456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698623.123378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134627584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698623.123387:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698623.123393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698623/real 1787698623] req@ffff9d85c0fb1500 x1874531134627584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698623.123399:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134627584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698623.123403:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698623.123405:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698623.123407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134627584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698623.123410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627712:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698623.123418:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698623.123423:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698623/real 1787698623] req@ffff9d85c0fb1880 x1874531134627712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698623.123429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134627712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698623.123434:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698623.123435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698623.123437:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698623.123440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698623.123448:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698623.123452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698623/real 1787698623] req@ffff9d85c0fb3100 x1874531134627840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698623.123458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134627840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698623.123462:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698623.123463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698623.123465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698623.123477:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698623.123493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698623.123501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698623/real 1787698623] req@ffff9d85c0fb1f80 x1874531134627968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698623.123511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134627968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698623.123518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698623.123520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698623.123523:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134627968:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698623.699075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698624.723010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698625.746988:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698626.770991:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698627.795089:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698628.242949:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698628.242957:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698628.242960:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698628.242967:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6530 00000100:00080000:2.0:1787698628.242976:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698628.242981:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698628.243009:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134628096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698628.243030:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698628.243034:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698628.243036:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698628.243040:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6530 00000100:00080000:2.0:1787698628.243044:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698628.243047:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698628.243056:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698628.243072:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134628224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698628.243084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698628.243092:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134628096:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698628.243105:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698628.243107:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698628.243112:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6530 00000100:00080000:2.0:1787698628.243116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698628.243120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698628.243126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698628.243142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134628352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698628.243151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698628.243153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698628.243154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698628.243157:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6530 00000100:00080000:2.0:1787698628.243160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698628.243162:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698628.243167:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698628.243173:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698628.243176:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134628480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698628.243183:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698628.243185:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698628.243187:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698628.243191:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6530 00000100:00080000:2.0:1787698628.243194:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698628.243198:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698628.243203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698628.243210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134628608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698628.243215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698628.243217:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698628.243218:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698628.243220:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6530 00000100:00080000:2.0:1787698628.243222:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698628.243225:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698628.243232:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698628.243234:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698628/real 1787698628] req@ffff9d85babd7800 x1874531134628096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698628.243264:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134628736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698628.243272:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134628096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698628.243288:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698628.243291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698628.243298:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698628.243299:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698628.243305:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698628.243306:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134628096:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698628.243318:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6530 00000100:00080000:2.0:1787698628.243330:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698628.243334:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698628.243341:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698628.243346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134628224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698628.243357:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134628864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698628.243375:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698628.243387:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698628/real 1787698628] req@ffff9d85babd6d80 x1874531134628224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698628.243395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134628224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698628.243401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698628.243404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698628.243406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134628224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698628.243410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134628352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698628.243422:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698628.243428:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698628/real 1787698628] req@ffff9d85babd5500 x1874531134628352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698628.243435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134628352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698628.243439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698628.243441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698628.243442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134628352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698628.243446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134628480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698628.243455:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698628.243460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698628/real 1787698628] req@ffff9d85babd6a00 x1874531134628480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698628.243467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134628480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698628.243472:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698628.243473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698628.243475:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134628480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698628.243478:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134628608:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698628.243487:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698628.243501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698628/real 1787698628] req@ffff9d85babd6680 x1874531134628608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698628.243508:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134628608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698628.243513:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698628.243515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698628.243517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134628608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698628.243520:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134628736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698628.243530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698628.243535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698628/real 1787698628] req@ffff9d85babd7480 x1874531134628736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698628.243541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134628736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698628.243546:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698628.243548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698628.243550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134628736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698628.243562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134628864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698628.243571:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698628.243576:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698628/real 1787698628] req@ffff9d85babd6300 x1874531134628864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698628.243582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134628864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698628.243587:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698628.243589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698628.243591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134628864:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698628.819053:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698629.843040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698630.867056:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698631.891056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698632.915023:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698633.362971:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698633.362981:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698633.362985:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698633.362997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6535 00000100:00080000:2.0:1787698633.363008:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698633.363016:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698633.363055:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134628992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698633.363087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698633.363091:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698633.363093:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698633.363097:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6535 00000100:00080000:2.0:1787698633.363100:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698633.363104:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698633.363113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698633.363127:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134629120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698633.363137:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134628992:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698633.363140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698633.363146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698633.363147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698633.363152:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6535 00000100:00080000:2.0:1787698633.363168:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698633.363173:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698633.363180:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698633.363193:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134629248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698633.363204:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698633.363207:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698633.363209:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698633.363214:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698633.363216:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6535 00000100:00080000:2.0:1787698633.363222:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698633.363227:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698633.363233:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698633.363244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134629376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698633.363253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698633.363257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698633.363259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698633.363267:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6535 00000100:00100000:1.0:1787698633.363268:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698633/real 1787698633] req@ffff9d85babd4380 x1874531134628992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698633.363272:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698633.363281:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698633.363288:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698633.363294:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134628992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698633.363303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134629504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698633.363306:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698633.363315:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698633.363323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134628992:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698633.363325:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698633.363330:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698633.363333:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698633.363340:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6535 00000100:00080000:2.0:1787698633.363345:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698633.363349:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134629120:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698633.363351:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698633.363360:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698633.363374:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134629632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698633.363379:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698633.363392:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698633.363394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698633/real 1787698633] req@ffff9d85babd4000 x1874531134629120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698633.363403:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698633.363408:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698633.363412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134629120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698633.363416:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6535 00000100:00080000:1.0:1787698633.363427:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698633.363427:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698633.363435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698633.363438:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698633.363441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134629120:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787698633.363447:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698633.363449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134629248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698633.363467:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134629760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698633.363471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698633.363483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698633/real 1787698633] req@ffff9d85babd5c00 x1874531134629248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698633.363492:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134629248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698633.363498:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698633.363500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698633.363503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134629248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698633.363508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134629376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698633.363518:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698633.363524:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698633/real 1787698633] req@ffff9d85babd4700 x1874531134629376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698633.363531:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134629376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698633.363536:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698633.363538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698633.363540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134629376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698633.363543:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134629504:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698633.363552:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698633.363557:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698633/real 1787698633] req@ffff9d85babd5f80 x1874531134629504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698633.363564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134629504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698633.363569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698633.363571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698633.363573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134629504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698633.363583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134629632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698633.363593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698633.363601:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698633/real 1787698633] req@ffff9d85babd7100 x1874531134629632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698633.363613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134629632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698633.363621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698633.363625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698633.363628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134629632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698633.363635:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134629760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698633.363655:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698633.363664:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698633/real 1787698633] req@ffff9d85babd4a80 x1874531134629760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698633.363672:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134629760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698633.363677:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698633.363679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698633.363682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134629760:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698633.939028:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698634.963062:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698635.987050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698637.011041:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698638.035051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698638.482989:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698638.483000:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698638.483004:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698638.483015:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6540 00000100:00080000:0.0:1787698638.483027:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698638.483035:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698638.483102:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134629888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698638.483125:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698638.483128:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698638.483129:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698638.483132:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6540 00000100:00080000:0.0:1787698638.483135:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698638.483138:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698638.483149:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698638.483158:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134630016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698638.483166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698638.483168:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698638.483169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698638.483173:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6540 00000100:00080000:0.0:1787698638.483176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698638.483177:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134629888:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698638.483179:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698638.483186:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698638.483192:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134630144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698638.483198:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698638.483200:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698638.483201:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698638.483203:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6540 00000100:00080000:0.0:1787698638.483204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698638.483207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698638.483209:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698638.483216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134630272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698638.483221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698638.483223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698638.483225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698638.483228:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6540 00000100:00080000:0.0:1787698638.483230:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698638.483233:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698638.483236:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698638.483242:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134630400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698638.483246:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698638.483248:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698638.483250:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698638.483252:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6540 00000100:00080000:0.0:1787698638.483255:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698638.483258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698638.483260:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698638.483268:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134630528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698638.483269:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698638.483276:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698638.483278:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698638.483279:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698638.483283:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6540 00000100:00080000:0.0:1787698638.483285:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698638.483288:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698638.483291:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698638.483299:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134630656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698638.483324:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698638/real 1787698638] req@ffff9d85b9d0b800 x1874531134629888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698638.483344:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134629888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698638.483355:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698638.483360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698638.483364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134629888:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698638.483396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698638.483420:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698638.483431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698638/real 1787698638] req@ffff9d85b9d09f80 x1874531134630016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698638.483442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134630016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698638.483451:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698638.483454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698638.483457:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698638.483461:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698638.483475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698638.483483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698638/real 1787698638] req@ffff9d85b9d0a300 x1874531134630144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698638.483500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134630144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698638.483507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698638.483510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698638.483513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698638.483518:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698638.483531:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698638.483539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698638/real 1787698638] req@ffff9d85b9d09180 x1874531134630272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698638.483549:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134630272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698638.483557:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698638.483559:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698638.483562:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698638.483566:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134630400:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698638.483579:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698638.483586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698638/real 1787698638] req@ffff9d85b9d09500 x1874531134630400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698638.483596:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134630400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698638.483603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698638.483606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698638.483608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134630400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698638.483613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134630528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698638.483625:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698638.483631:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698638/real 1787698638] req@ffff9d85b9d08700 x1874531134630528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698638.483640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134630528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698638.483647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698638.483649:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698638.483652:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134630528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698638.483656:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134630656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698638.483668:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698638.483675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698638/real 1787698638] req@ffff9d85b9d09c00 x1874531134630656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698638.483684:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134630656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698638.483691:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698638.483694:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698638.483697:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134630656:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698639.059104:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698640.083067:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698641.107128:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698642.131042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698643.155080:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698643.603033:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698643.603049:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698643.603053:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698643.603066:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6545 00000100:00080000:2.0:1787698643.603080:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698643.603089:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698643.603142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134630784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698643.603174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698643.603178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698643.603180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698643.603185:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6545 00000100:00080000:2.0:1787698643.603188:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698643.603193:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698643.603205:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698643.603218:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134630912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698643.603224:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134630784:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698643.603234:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698643.603238:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698643.603240:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698643.603244:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6545 00000100:00080000:2.0:1787698643.603247:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698643.603251:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698643.603256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698643.603268:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134631040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698643.603277:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698643.603280:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698643.603282:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698643.603286:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6545 00000100:00080000:2.0:1787698643.603289:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698643.603293:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698643.603298:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698643.603309:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134631168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698643.603318:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698643.603321:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698643.603323:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698643.603326:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6545 00000100:00080000:2.0:1787698643.603330:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698643.603334:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698643.603339:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698643.603348:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134631296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698643.603357:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698643.603361:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698643.603363:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698643.603367:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6545 00000100:00080000:2.0:1787698643.603372:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698643.603376:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698643.603382:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698643.603383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698643.603407:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134631424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698643.603419:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698643.603423:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631040:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698643.603425:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698643.603429:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698643.603446:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6545 00000100:00080000:2.0:1787698643.603451:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698643.603458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631168:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698643.603458:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698643.603464:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698643.603480:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134631552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698643.603492:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134631296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698643.603515:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134631424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698643.603539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134631552:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698644.179088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698645.203042:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698646.227014:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698646.675161:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:3.0:1787698646.675199:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698646.675214:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698646.675261:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698646.675272:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698646.675280:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698646.675287:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698646.675294:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698646.675320:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698646.675422:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698643/real 1787698646] req@ffff9d85babd5f80 x1874531134631424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698646.675459:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134631424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698646.675470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698646.675475:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698646.675480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134631424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698646.675489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698643/real 1787698646] req@ffff9d85babd4700 x1874531134631552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698646.675502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134631552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698646.675517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698646.675521:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698646.675525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134631552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698646.675532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698643/real 1787698646] req@ffff9d85babd4e00 x1874531134630912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698646.675543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134630912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698646.675551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698646.675554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698646.675557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134630912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698646.675563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698643/real 1787698646] req@ffff9d85babd7b80 x1874531134631040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698646.675574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134631040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698646.675580:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698646.675584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698646.675587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698646.675592:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698643/real 1787698646] req@ffff9d85babd4a80 x1874531134631168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698646.675603:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134631168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698646.675610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698646.675613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698646.675616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698646.675624:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698643/real 1787698646] req@ffff9d85babd7100 x1874531134631296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698646.675642:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134631296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698646.675650:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698646.675653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698646.675657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134631296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698646.675664:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698643/real 1787698646] req@ffff9d85babd5180 x1874531134630784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698646.675677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134630784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698646.675685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698646.675688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698646.675691:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134630784:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787698647.251042:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698648.275033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698648.723039:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698648.723049:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698648.723053:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698648.723063:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6550 00000100:00080000:3.0:1787698648.723075:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698648.723082:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698648.723118:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134631680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698648.723144:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698648.723147:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698648.723148:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698648.723155:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6550 00000100:00080000:3.0:1787698648.723157:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698648.723160:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698648.723169:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698648.723179:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134631808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698648.723185:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698648.723187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698648.723188:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698648.723191:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6550 00000100:00080000:3.0:1787698648.723192:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698648.723195:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698648.723198:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698648.723207:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134631936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698648.723208:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134631680:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698648.723221:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698648.723224:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698648.723226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698648.723230:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6550 00000100:00080000:3.0:1787698648.723234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698648.723238:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698648.723243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698648.723256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134632064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698648.723266:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698648.723281:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698648.723286:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698648.723291:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6550 00000100:00080000:3.0:1787698648.723295:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698648.723300:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698648.723306:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787698648.723309:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698648.723325:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134632192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698648.723334:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698648.723338:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698648.723340:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698648.723344:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6550 00000100:00080000:3.0:1787698648.723349:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698648.723353:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698648.723358:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698648.723369:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134632320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698648.723383:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698648/real 1787698648] req@ffff9d85c0fb1c00 x1874531134631680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698648.723386:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698648.723395:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698648.723399:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698648.723408:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6550 00000100:00080000:3.0:1787698648.723413:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698648.723419:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134631680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698648.723419:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698648.723428:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698648.723439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698648.723446:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134632448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698648.723448:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698648.723458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134631680:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698648.723469:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631808:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698648.723496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698648.723510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698648/real 1787698648] req@ffff9d85c0fb0a80 x1874531134631808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698648.723523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134631808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698648.723532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698648.723535:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698648.723539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698648.723566:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698648.723585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698648.723596:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698648/real 1787698648] req@ffff9d85c0fb3b80 x1874531134631936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698648.723607:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134631936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698648.723621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698648.723625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698648.723628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134631936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698648.723633:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698648.723648:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698648.723657:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698648/real 1787698648] req@ffff9d85c0fb0380 x1874531134632064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698648.723667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134632064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698648.723675:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698648.723679:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698648.723682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698648.723688:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134632192:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698648.723705:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698648.723715:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698648/real 1787698648] req@ffff9d85c0fb2d80 x1874531134632192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698648.723728:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134632192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698648.723735:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698648.723744:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698648.723749:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134632192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698648.723755:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134632320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698648.723773:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698648.723782:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698648/real 1787698648] req@ffff9d85c0fb3800 x1874531134632320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698648.723792:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134632320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698648.723800:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698648.723802:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698648.723806:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134632320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698648.723811:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134632448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698648.723826:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698648.723835:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698648/real 1787698648] req@ffff9d85c0fb0000 x1874531134632448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698648.723845:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134632448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698648.723884:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698648.723888:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698648.723891:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134632448:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698649.299091:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698650.323049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698651.347077:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698652.371097:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698653.394992:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698653.843010:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698653.843022:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698653.843027:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698653.843040:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6555 00000100:00080000:2.0:1787698653.843055:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698653.843064:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698653.843110:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134632576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698653.843141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698653.843144:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698653.843146:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698653.843150:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6555 00000100:00080000:2.0:1787698653.843154:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698653.843157:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698653.843168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698653.843181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134632704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698653.843191:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698653.843194:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134632576:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698653.843195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698653.843197:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698653.843205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6555 00000100:00080000:2.0:1787698653.843209:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698653.843212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698653.843217:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698653.843228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134632832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698653.843237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698653.843241:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698653.843243:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698653.843246:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6555 00000100:00080000:2.0:1787698653.843250:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698653.843254:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698653.843258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698653.843267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134632960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698653.843276:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698653.843277:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698653.843282:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698653.843284:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698653.843289:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6555 00000100:00080000:2.0:1787698653.843292:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698653.843296:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698653.843301:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698653.843311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134633088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698653.843318:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698653.843322:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698653.843324:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698653.843327:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6555 00000100:00080000:2.0:1787698653.843330:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698653.843334:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698653.843339:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698653.843344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698653/real 1787698653] req@ffff9d85babd5c00 x1874531134632576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698653.843351:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134633216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698653.843362:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698653.843366:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134632576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698653.843368:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698653.843372:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698653.843381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698653.843381:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6555 00000100:00080000:1.0:1787698653.843388:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698653.843388:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698653.843395:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134632576:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698653.843397:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698653.843404:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698653.843416:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134633344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698653.843421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632704:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698653.843444:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698653.843454:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698653/real 1787698653] req@ffff9d85babd4000 x1874531134632704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698653.843461:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134632704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698653.843467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698653.843469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698653.843471:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698653.843474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698653.843484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698653.843489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698653/real 1787698653] req@ffff9d85babd4380 x1874531134632832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698653.843498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134632832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698653.843511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698653.843514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698653.843517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698653.843523:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698653.843542:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698653.843552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698653/real 1787698653] req@ffff9d85babd6300 x1874531134632960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698653.843559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134632960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698653.843564:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698653.843566:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698653.843568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134632960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698653.843572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698653.843583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698653.843589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698653/real 1787698653] req@ffff9d85babd7480 x1874531134633088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698653.843596:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134633088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698653.843600:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698653.843601:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698653.843604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698653.843606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698653.843615:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698653.843620:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698653/real 1787698653] req@ffff9d85babd6680 x1874531134633216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698653.843625:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134633216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698653.843630:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698653.843632:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698653.843633:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698653.843644:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698653.843653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698653.843658:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698653/real 1787698653] req@ffff9d85babd6a00 x1874531134633344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698653.843663:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134633344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698653.843668:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698653.843670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698653.843671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633344:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698654.419053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698655.443065:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698656.467064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698657.491066:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698658.515049:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698658.963008:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698658.963017:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698658.963022:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698658.963032:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6560 00000100:00080000:3.0:1787698658.963043:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698658.963049:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698658.963087:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134633472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698658.963111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698658.963113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698658.963115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698658.963118:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6560 00000100:00080000:3.0:1787698658.963121:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698658.963123:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698658.963133:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698658.963141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134633600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698658.963148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698658.963153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698658.963154:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134633472:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698658.963154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698658.963162:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6560 00000100:00080000:3.0:1787698658.963164:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698658.963167:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698658.963170:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698658.963176:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134633728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698658.963181:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698658.963184:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698658.963185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698658.963187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6560 00000100:00080000:3.0:1787698658.963189:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698658.963192:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698658.963196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698658.963210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134633856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698658.963238:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698658.963290:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698658/real 1787698658] req@ffff9d85c0fb0e00 x1874531134633472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.963310:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134633472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.963318:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.963325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698658.963335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134633472:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698658.963369:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698658.963370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134633600:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698658.963382:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698658.963388:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698658.963399:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6560 00000800:00000100:1.0:1787698658.963409:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698658.963414:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698658.963421:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698658/real 1787698658] req@ffff9d85c0fb2680 x1874531134633600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698658.963426:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698658.963442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134633600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698658.963445:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698658.963467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.963473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698658.963476:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134633600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698658.963487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134633728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787698658.963489:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134633984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698658.963507:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698658.963510:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698658.963513:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698658.963520:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698658.963524:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6560 00000100:00080000:3.0:1787698658.963529:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698658.963533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698658/real 1787698658] req@ffff9d85c0fb2a00 x1874531134633728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698658.963534:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698658.963547:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698658.963560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134633728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698658.963570:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134634112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698658.963579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.963589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698658.963594:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698658.963600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134633728:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698658.963602:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698658.963606:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698658.963614:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134633856:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787698658.963614:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6560 00000100:00080000:3.0:1787698658.963618:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698658.963625:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698658.963634:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698658.963644:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698658.963652:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134634240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698658.963656:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698658/real 1787698658] req@ffff9d85c0fb0000 x1874531134633856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.963689:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134633856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.963705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.963710:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698658.963713:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134633856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698658.963755:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633984:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698658.963771:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698658.963781:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698658/real 1787698658] req@ffff9d85c0fb3800 x1874531134633984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.963792:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134633984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.963800:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.963803:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698658.963807:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134633984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698658.963814:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134634112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698658.963830:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698658.963838:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698658/real 1787698658] req@ffff9d85c0fb2d80 x1874531134634112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.963845:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134634112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.963871:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.963873:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698658.963875:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134634112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698658.963879:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134634240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698658.963891:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698658.963898:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698658/real 1787698658] req@ffff9d85c0fb0380 x1874531134634240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.963904:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134634240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.963909:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.963911:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698658.963913:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134634240:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698659.539038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698660.563171:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698661.587063:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698662.611081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698663.634965:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698664.082970:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698664.082981:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698664.082985:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698664.082995:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6565 00000100:00080000:3.0:1787698664.083007:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698664.083014:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698664.083065:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134634368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698664.083101:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698664.083106:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698664.083108:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698664.083112:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6565 00000100:00080000:3.0:1787698664.083117:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698664.083129:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698664.083142:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698664.083153:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134634368:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698664.083155:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134634496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698664.083168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698664.083171:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698664.083173:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698664.083175:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6565 00000100:00080000:3.0:1787698664.083178:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698664.083180:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698664.083184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698664.083191:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134634624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698664.083198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698664.083201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698664.083202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698664.083204:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6565 00000100:00080000:3.0:1787698664.083207:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698664.083209:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698664.083211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698664.083217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134634752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698664.083223:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698664.083225:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698664.083226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698664.083228:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6565 00000100:00080000:3.0:1787698664.083231:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698664.083235:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698664.083237:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698664.083242:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698664.083249:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134634880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698664.083256:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698664.083258:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698664.083260:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698664.083262:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6565 00000100:00080000:3.0:1787698664.083263:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698664.083266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698664.083268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698664.083275:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134635008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698664.083279:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698664.083281:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698664.083282:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698664.083285:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6565 00000100:00080000:3.0:1787698664.083289:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698664.083293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698664/real 1787698664] req@ffff9d85c0fb3b80 x1874531134634368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698664.083297:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698664.083307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698664.083319:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134634368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698664.083325:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134635136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698664.083330:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698664.083338:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698664.083343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134634368:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698664.083367:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134634496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698664.083382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698664.083390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698664/real 1787698664] req@ffff9d85c0fb0a80 x1874531134634496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698664.083397:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134634496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698664.083402:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698664.083404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698664.083406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134634496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698664.083409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134634624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698664.083419:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698664.083424:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698664/real 1787698664] req@ffff9d85c0fb1c00 x1874531134634624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698664.083431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134634624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698664.083439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698664.083441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698664.083443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134634624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698664.083446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134634752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698664.083454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698664.083459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698664/real 1787698664] req@ffff9d85c0fb1f80 x1874531134634752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698664.083464:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134634752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698664.083469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698664.083471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698664.083473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134634752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698664.083475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134634880:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698664.083484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698664.083489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698664/real 1787698664] req@ffff9d85c0fb3100 x1874531134634880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698664.083494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134634880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698664.083498:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698664.083500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698664.083502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134634880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698664.083506:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698664.083515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698664.083520:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698664/real 1787698664] req@ffff9d85c0fb1880 x1874531134635008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698664.083526:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134635008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698664.083530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698664.083532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698664.083533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698664.083536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698664.083544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698664.083548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698664/real 1787698664] req@ffff9d85c0fb1500 x1874531134635136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698664.083553:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134635136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698664.083558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698664.083560:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698664.083562:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635136:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698664.659026:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698665.683034:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698666.707065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698667.731071:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698668.755006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698669.202940:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698669.202952:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698669.202958:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698669.202972:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6570 00000100:00080000:3.0:1787698669.202985:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698669.202994:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698669.203041:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134635264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698669.203072:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698669.203076:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698669.203078:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698669.203083:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6570 00000100:00080000:3.0:1787698669.203087:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698669.203091:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698669.203102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698669.203115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134635392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698669.203126:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134635264:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698669.203128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698669.203136:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698669.203139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698669.203143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6570 00000100:00080000:3.0:1787698669.203146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698669.203150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698669.203156:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698669.203167:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134635520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698669.203183:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698669.203187:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698669.203189:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698669.203192:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6570 00000100:00080000:3.0:1787698669.203196:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698669.203207:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698669.203212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698669.203214:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698669.203229:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134635648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698669.203239:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698669.203243:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698669.203244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698669.203248:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6570 00000100:00080000:3.0:1787698669.203251:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698669.203255:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698669.203260:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698669.203272:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698669/real 1787698669] req@ffff9d85c0fb2300 x1874531134635264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698669.203273:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134635776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698669.203298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698669.203303:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698669.203305:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698669.203311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134635264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698669.203312:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6570 00000100:00080000:3.0:1787698669.203318:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698669.203326:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698669.203331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698669.203332:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698669.203341:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:3.0:1787698669.203347:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134635904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698669.203350:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134635264:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698669.203364:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698669.203368:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698669.203370:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698669.203375:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6570 00000100:00080000:3.0:1787698669.203378:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698669.203385:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134635392:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698669.203386:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698669.203394:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698669.203417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698669.203422:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134636032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698669.203434:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698669/real 1787698669] req@ffff9d85c0fb3480 x1874531134635392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698669.203452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134635392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698669.203461:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698669.203465:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698669.203468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134635392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698669.203475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134635520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698669.203497:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698669.203508:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698669/real 1787698669] req@ffff9d85c0fb0700 x1874531134635520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698669.203520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134635520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698669.203534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698669.203538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698669.203541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134635520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698669.203574:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134635648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698669.203607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698669.203618:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698669/real 1787698669] req@ffff9d85c0fb1500 x1874531134635648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698669.203629:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134635648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698669.203637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698669.203640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698669.203643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134635648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698669.203648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635776:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698669.203667:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698669.203677:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698669/real 1787698669] req@ffff9d85c0fb1880 x1874531134635776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698669.203688:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134635776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698669.203696:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698669.203699:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698669.203702:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698669.203709:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698669.203727:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698669.203738:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698669/real 1787698669] req@ffff9d85c0fb3100 x1874531134635904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698669.203748:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134635904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698669.203757:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698669.203760:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698669.203763:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134635904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698669.203780:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698669.203799:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698669.203809:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698669/real 1787698669] req@ffff9d85c0fb1f80 x1874531134636032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698669.203820:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134636032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698669.203828:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698669.203831:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698669.203834:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636032:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698669.779076:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698670.803009:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698671.827081:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698672.851081:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698673.875046:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698674.322960:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698674.322967:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698674.322970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698674.322977:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6576 00000100:00080000:2.0:1787698674.322986:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698674.322991:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698674.323018:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134636160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698674.323037:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698674.323039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698674.323040:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698674.323043:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6576 00000100:00080000:2.0:1787698674.323045:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698674.323047:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698674.323055:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698674.323067:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134636288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698674.323074:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134636160:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698674.323081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698674.323086:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698674.323088:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698674.323092:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6576 00000100:00080000:2.0:1787698674.323096:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698674.323101:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698674.323107:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698674.323126:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698674.323129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134636416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698674.323141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698674.323144:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698674.323146:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698674.323150:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6576 00000100:00080000:2.0:1787698674.323154:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698674.323161:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698674/real 1787698674] req@ffff9d85babd5500 x1874531134636160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698674.323161:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698674.323168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698674.323183:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134636160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698674.323190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134636544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698674.323201:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698674.323206:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698674.323208:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698674.323213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787698674.323214:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134636160:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698674.323218:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698674.323224:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6576 00000100:00080000:2.0:1787698674.323228:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698674.323232:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698674.323237:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134636288:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787698674.323238:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698674.323254:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134636672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698674.323261:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698674.323271:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698674.323275:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698674.323278:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698674/real 1787698674] req@ffff9d85babd6d80 x1874531134636288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698674.323279:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698674.323286:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6576 00000100:00080000:2.0:1787698674.323292:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698674.323297:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698674.323302:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134636288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698674.323311:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698674.323314:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698674.323318:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698674.323320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134636288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698674.323327:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134636416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698674.323328:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134636800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698674.323340:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698674.323344:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698674.323345:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698674.323349:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698674.323356:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6576 00000100:00100000:1.0:1787698674.323357:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698674/real 1787698674] req@ffff9d85babd7800 x1874531134636416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698674.323363:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698674.323369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134636416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698674.323371:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698674.323384:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698674.323386:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698674.323387:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698674.323392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134636416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698674.323397:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134636544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698674.323404:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134636928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698674.323411:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698674.323420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698674/real 1787698674] req@ffff9d85babd6a00 x1874531134636544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698674.323427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134636544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698674.323433:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698674.323435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698674.323438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134636544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698674.323452:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636672:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698674.323482:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698674.323493:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698674/real 1787698674] req@ffff9d85babd6680 x1874531134636672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698674.323501:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134636672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698674.323506:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698674.323507:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698674.323510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698674.323513:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698674.323525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698674.323530:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698674/real 1787698674] req@ffff9d85babd7480 x1874531134636800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698674.323537:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134636800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698674.323541:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698674.323542:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698674.323544:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698674.323546:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698674.323554:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698674.323559:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698674/real 1787698674] req@ffff9d85babd6300 x1874531134636928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698674.323565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134636928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698674.323569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698674.323570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698674.323572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134636928:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698674.899048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698675.923006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698676.947007:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698677.971034:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698678.994988:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698679.442945:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698679.442955:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698679.442959:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698679.442969:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6581 00000100:00080000:2.0:1787698679.442979:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698679.442987:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698679.443016:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134637056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698679.443036:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698679.443040:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698679.443041:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698679.443045:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6581 00000100:00080000:2.0:1787698679.443049:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698679.443052:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698679.443061:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698679.443073:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134637184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698679.443080:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134637056:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698679.443082:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698679.443085:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698679.443086:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698679.443090:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6581 00000100:00080000:2.0:1787698679.443093:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698679.443095:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698679.443108:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698679.443116:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134637312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698679.443123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698679.443125:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698679.443127:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698679.443130:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6581 00000800:00000100:1.0:1787698679.443133:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698679.443133:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698679.443136:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698679.443139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698679.443147:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134637440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698679.443154:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698679.443157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698679.443159:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698679.443161:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6581 00000100:00080000:2.0:1787698679.443164:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698679.443167:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698679.443169:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698679/real 1787698679] req@ffff9d85babd4380 x1874531134637056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698679.443171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698679.443178:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134637568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698679.443186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698679.443188:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698679.443190:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698679.443192:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134637056/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698679.443193:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6581 00000100:00080000:2.0:1787698679.443196:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698679.443200:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698679.443202:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698679.443203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698679.443205:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698679.443208:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134637056:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698679.443211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134637696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698679.443218:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698679.443220:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698679.443222:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698679.443224:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6581 00000100:00080000:2.0:1787698679.443227:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698679.443228:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134637184:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698679.443230:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698679.443234:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698679.443243:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134637824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698679.443245:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698679.443255:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698679/real 1787698679] req@ffff9d85babd4000 x1874531134637184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698679.443265:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134637184/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698679.443273:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698679.443276:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698679.443279:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134637184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698679.443285:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134637312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698679.443301:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698679.443308:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698679/real 1787698679] req@ffff9d85babd5c00 x1874531134637312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698679.443315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134637312/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698679.443323:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698679.443325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698679.443327:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134637312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698679.443330:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134637440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698679.443339:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698679.443344:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698679/real 1787698679] req@ffff9d85babd5180 x1874531134637440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698679.443350:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134637440/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698679.443354:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698679.443356:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698679.443357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134637440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698679.443360:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134637568:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698679.443389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698679.443397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698679/real 1787698679] req@ffff9d85babd7100 x1874531134637568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698679.443403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134637568/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698679.443408:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698679.443410:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698679.443412:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134637568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698679.443415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134637696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698679.443429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698679.443437:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698679/real 1787698679] req@ffff9d85babd4a80 x1874531134637696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698679.443448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134637696/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698679.443455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698679.443458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698679.443461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134637696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698679.443477:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134637824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698679.443491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698679.443500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698679/real 1787698679] req@ffff9d85babd7b80 x1874531134637824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698679.443510:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134637824/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698679.443517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698679.443520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698679.443523:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134637824:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698680.019021:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698681.043022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698682.067024:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698683.091016:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698684.114988:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698684.562943:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698684.562950:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698684.562953:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698684.562961:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6586 00000100:00080000:2.0:1787698684.562970:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698684.562975:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698684.563004:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134637952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698684.563025:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698684.563027:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698684.563028:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698684.563031:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6586 00000100:00080000:2.0:1787698684.563033:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698684.563036:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698684.563043:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698684.563050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134638080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698684.563056:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698684.563058:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698684.563060:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698684.563081:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134637952:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698684.563089:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6586 00000100:00080000:2.0:1787698684.563093:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698684.563097:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698684.563102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698684.563115:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134638208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698684.563124:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698684.563128:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698684.563130:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698684.563133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6586 00000100:00080000:2.0:1787698684.563137:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698684.563140:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698684.563144:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698684.563148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698684.563160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134638336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698684.563166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698684.563168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698684.563169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698684.563172:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6586 00000100:00080000:2.0:1787698684.563174:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698684.563176:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698684.563178:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698684.563185:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134638464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698684.563192:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698684/real 1787698684] req@ffff9d85babd4e00 x1874531134637952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698684.563193:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698684.563199:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698684.563202:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698684.563209:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6586 00000100:00080000:2.0:1787698684.563214:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698684.563216:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698684.563219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698684.563223:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134637952/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698684.563228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134638592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698684.563244:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698684.563246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698684.563251:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698684.563253:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698684.563254:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698684.563261:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134637952:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698684.563263:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6586 00000100:00080000:2.0:1787698684.563271:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698684.563276:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638080:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698684.563278:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698684.563298:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698684.563311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134638720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698684.563330:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698684.563346:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698684/real 1787698684] req@ffff9d85babd4700 x1874531134638080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698684.563358:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134638080/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698684.563366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698684.563370:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698684.563373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698684.563394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698684.563412:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698684.563422:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698684/real 1787698684] req@ffff9d85babd5f80 x1874531134638208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698684.563433:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134638208/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698684.563447:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698684.563450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698684.563453:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698684.563459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698684.563474:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698684.563483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698684/real 1787698684] req@ffff9d85babd7b80 x1874531134638336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698684.563494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134638336/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698684.563502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698684.563505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698684.563508:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698684.563513:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134638464:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698684.563530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698684.563539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698684/real 1787698684] req@ffff9d85babd4a80 x1874531134638464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698684.563551:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134638464/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698684.563558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698684.563561:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698684.563564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134638464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698684.563570:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134638592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698684.563584:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698684.563593:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698684/real 1787698684] req@ffff9d85babd7100 x1874531134638592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698684.563604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134638592/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698684.563612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698684.563614:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698684.563617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134638592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698684.563623:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134638720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698684.563637:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698684.563645:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698684/real 1787698684] req@ffff9d85babd5180 x1874531134638720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698684.563656:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134638720/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698684.563663:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698684.563666:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698684.563669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134638720:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698685.138988:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698686.163000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698687.187025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698688.210998:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698689.235019:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698689.682983:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698689.682992:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698689.682996:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698689.683007:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6591 00000100:00080000:3.0:1787698689.683016:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698689.683022:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698689.683055:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134638848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698689.683081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698689.683084:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698689.683086:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698689.683090:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6591 00000100:00080000:3.0:1787698689.683093:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698689.683096:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698689.683105:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698689.683116:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134638976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698689.683125:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698689.683129:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698689.683132:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698689.683133:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134638848:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698689.683138:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6591 00000100:00080000:3.0:1787698689.683143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698689.683146:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698689.683151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698689.683162:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134639104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698689.683176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698689.683179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698689.683181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698689.683184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6591 00000100:00080000:3.0:1787698689.683188:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698689.683192:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698689.683207:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698689.683215:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698689.683220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134639232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698689.683230:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698689.683233:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698689.683235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698689.683238:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6591 00000100:00080000:3.0:1787698689.683242:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698689.683246:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698689.683250:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698689.683260:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134639360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698689.683265:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698689/real 1787698689] req@ffff9d85c0fb1c00 x1874531134638848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698689.683275:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698689.683284:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698689.683287:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698689.683291:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6591 00000100:00080000:3.0:1787698689.683294:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698689.683297:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134638848/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698689.683301:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698689.683307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698689.683314:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698689.683320:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134639488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698689.683323:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698689.683330:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134638848:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698689.683344:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698689.683347:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698689.683349:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698689.683353:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6591 00000100:00080000:3.0:1787698689.683356:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698689.683358:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638976:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698689.683363:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698689.683368:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698689.683380:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134639616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698689.683389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698689.683404:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698689/real 1787698689] req@ffff9d85c0fb0a80 x1874531134638976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698689.683416:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134638976/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698689.683424:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698689.683427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698689.683430:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134638976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698689.683435:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134639104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698689.683451:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698689.683460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698689/real 1787698689] req@ffff9d85c0fb3b80 x1874531134639104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698689.683470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134639104/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698689.683482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698689.683486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698689.683488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134639104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698689.683494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134639232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698689.683509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698689.683518:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698689/real 1787698689] req@ffff9d85c0fb0380 x1874531134639232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698689.683527:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134639232/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698689.683535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698689.683538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698689.683541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134639232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698689.683546:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134639360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698689.683560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698689.683569:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698689/real 1787698689] req@ffff9d85c0fb2d80 x1874531134639360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698689.683579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134639360/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698689.683586:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698689.683589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698689.683592:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134639360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698689.683597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134639488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698689.683612:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698689.683621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698689/real 1787698689] req@ffff9d85c0fb3800 x1874531134639488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698689.683631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134639488/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698689.683638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698689.683640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698689.683643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134639488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698689.683659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134639616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698689.683676:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698689.683686:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698689/real 1787698689] req@ffff9d85c0fb0000 x1874531134639616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698689.683696:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134639616/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698689.683702:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698689.683705:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698689.683708:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134639616:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698690.258978:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698691.283024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698692.306963:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698693.331046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698694.355008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698694.802938:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698694.802946:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698694.802954:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698694.802961:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6596 00000100:00080000:3.0:1787698694.802970:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698694.802976:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698694.803003:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134639744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698694.803020:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698694.803022:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698694.803024:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698694.803026:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6596 00000100:00080000:3.0:1787698694.803028:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698694.803031:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698694.803037:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698694.803045:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134639872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698694.803050:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698694.803053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698694.803054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698694.803056:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6596 00000100:00080000:3.0:1787698694.803057:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698694.803061:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134639744:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698694.803062:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698694.803068:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698694.803076:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134640000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698694.803081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698694.803084:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698694.803086:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698694.803089:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6596 00000100:00080000:3.0:1787698694.803093:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698694.803096:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698694.803100:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698694.803127:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698694.803137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134640128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698694.803147:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698694.803149:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698694.803151:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698694.803153:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6596 00000100:00080000:3.0:1787698694.803156:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698694.803159:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698694.803162:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698694.803167:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698694/real 1787698694] req@ffff9d85c0fb2a00 x1874531134639744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698694.803172:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134640256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698694.803182:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698694.803186:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134639744/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698694.803186:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698694.803188:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698694.803194:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6596 00000100:00080000:3.0:1787698694.803198:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698694.803201:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698694.803202:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698694.803204:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698694.803207:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698694.803211:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134639744:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698694.803214:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134640384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698694.803221:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134639872:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698694.803222:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698694.803227:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698694.803228:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698694.803231:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6596 00000100:00080000:3.0:1787698694.803233:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698694.803236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698694.803239:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698694.803243:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698694.803248:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134640512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698694.803254:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698694/real 1787698694] req@ffff9d85c0fb2680 x1874531134639872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698694.803262:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134639872/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698694.803268:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698694.803270:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698694.803272:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134639872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698694.803288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698694.803298:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698694.803304:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698694/real 1787698694] req@ffff9d85c0fb0e00 x1874531134640000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698694.803310:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134640000/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698694.803327:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698694.803329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698694.803331:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698694.803335:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698694.803344:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698694.803350:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698694/real 1787698694] req@ffff9d85c0fb0000 x1874531134640128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698694.803356:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134640128/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698694.803360:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698694.803362:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698694.803363:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698694.803366:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134640256:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698694.803375:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698694.803380:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698694/real 1787698694] req@ffff9d85c0fb3800 x1874531134640256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698694.803386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134640256/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698694.803390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698694.803391:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698694.803394:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134640256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698694.803397:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134640384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698694.803405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698694.803410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698694/real 1787698694] req@ffff9d85c0fb2d80 x1874531134640384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698694.803416:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134640384/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698694.803420:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698694.803422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698694.803423:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134640384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698694.803426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134640512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698694.803433:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698694.803438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698694/real 1787698694] req@ffff9d85c0fb0380 x1874531134640512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698694.803443:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134640512/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698694.803447:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698694.803449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698694.803451:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134640512:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698695.379022:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698696.403021:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698697.427024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698698.451014:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698699.475004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698699.922960:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698699.922970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698699.922974:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698699.922984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6601 00000100:00080000:3.0:1787698699.922993:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698699.922999:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698699.923050:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134640640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698699.923070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698699.923072:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698699.923073:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698699.923076:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6601 00000100:00080000:3.0:1787698699.923078:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698699.923081:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698699.923087:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698699.923095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134640768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698699.923101:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698699.923103:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698699.923104:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698699.923106:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6601 00000100:00080000:3.0:1787698699.923108:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698699.923113:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698699.923115:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134640640:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698699.923117:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698699.923126:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134640896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698699.923132:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698699.923134:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698699.923136:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698699.923138:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6601 00000100:00080000:3.0:1787698699.923139:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698699.923141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698699.923144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698699.923151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134641024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698699.923159:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698699.923162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698699.923163:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698699.923166:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6601 00000100:00080000:3.0:1787698699.923169:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698699.923173:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698699.923177:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698699.923187:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134641152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698699.923201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698699.923204:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698699.923205:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698699.923209:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6601 00000100:00080000:3.0:1787698699.923213:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698699.923214:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698699.923221:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698699.923232:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698699.923243:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134641280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698699.923250:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698699.923252:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698699.923254:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698699.923256:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6601 00000100:00080000:3.0:1787698699.923258:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698699.923261:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698699.923265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698699.923269:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698699/real 1787698699] req@ffff9d85c0fb3b80 x1874531134640640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698699.923274:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134641408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698699.923304:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134640640/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698699.923315:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698699.923318:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698699.923322:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134640640:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698699.923329:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640768:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698699.923350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698699.923360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698699/real 1787698699] req@ffff9d85c0fb0a80 x1874531134640768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698699.923370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134640768/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698699.923377:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698699.923379:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698699.923382:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698699.923403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698699.923430:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698699.923442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698699/real 1787698699] req@ffff9d85c0fb1c00 x1874531134640896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698699.923453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134640896/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698699.923467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698699.923470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698699.923473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134640896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698699.923479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698699.923496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698699.923507:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698699/real 1787698699] req@ffff9d85c0fb1f80 x1874531134641024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698699.923518:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134641024/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698699.923526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698699.923529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698699.923532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698699.923538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134641152:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698699.923557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698699.923565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698699/real 1787698699] req@ffff9d85c0fb3100 x1874531134641152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698699.923572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134641152/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698699.923583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698699.923585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698699.923588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134641152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698699.923593:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134641280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698699.923605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698699.923611:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698699/real 1787698699] req@ffff9d85c0fb1880 x1874531134641280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698699.923617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134641280/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698699.923621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698699.923623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698699.923624:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134641280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698699.923627:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134641408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698699.923636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698699.923640:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698699/real 1787698699] req@ffff9d85c0fb1500 x1874531134641408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698699.923647:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134641408/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698699.923651:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698699.923652:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698699.923654:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134641408:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698700.499076:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698701.523008:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698702.547005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698703.571035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698704.594970:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698705.042953:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698705.042963:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698705.042967:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698705.042976:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6606 00000100:00080000:3.0:1787698705.042986:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698705.042993:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698705.043028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134641536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698705.043054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698705.043058:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698705.043059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698705.043063:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6606 00000100:00080000:3.0:1787698705.043066:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698705.043070:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698705.043079:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698705.043092:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134641664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698705.043095:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134641536:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698705.043110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698705.043113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698705.043115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698705.043119:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6606 00000100:00080000:3.0:1787698705.043123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698705.043126:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698705.043132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698705.043141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134641792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698705.043150:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698705.043155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698705.043156:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698705.043158:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698705.043165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6606 00000100:00080000:3.0:1787698705.043169:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698705.043173:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698705.043178:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698705.043189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134641920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698705.043198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698705.043201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698705.043203:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698705.043208:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698705/real 1787698705] req@ffff9d85c0fb0700 x1874531134641536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698705.043209:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6606 00000100:00080000:3.0:1787698705.043221:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698705.043225:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134641536/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698705.043227:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698705.043235:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698705.043237:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698705.043242:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698705.043245:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134641536:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698705.043250:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134642048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698705.043259:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698705.043262:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698705.043264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698705.043265:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641664:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698705.043271:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6606 00000100:00080000:3.0:1787698705.043275:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698705.043279:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698705.043285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698705.043293:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698705.043297:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134642176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698705.043307:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698705.043311:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698705/real 1787698705] req@ffff9d85c0fb3480 x1874531134641664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698705.043312:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698705.043315:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698705.043322:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6606 00000100:00080000:3.0:1787698705.043328:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698705.043333:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698705.043337:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134641664/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698705.043338:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698705.043348:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698705.043352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134642304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698705.043353:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698705.043359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698705.043369:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698705.043392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698705.043403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698705/real 1787698705] req@ffff9d85c0fb2300 x1874531134641792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698705.043414:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134641792/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698705.043426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698705.043428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698705.043430:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698705.043434:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698705.043450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698705.043459:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698705/real 1787698705] req@ffff9d85c0fb1500 x1874531134641920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698705.043470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134641920/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698705.043478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698705.043481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698705.043484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134641920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698705.043489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642048:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698705.043505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698705.043512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698705/real 1787698705] req@ffff9d85c0fb1880 x1874531134642048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698705.043519:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134642048/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698705.043523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698705.043525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698705.043529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642048:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698705.043544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698705.043561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698705.043572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698705/real 1787698705] req@ffff9d85c0fb3100 x1874531134642176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698705.043583:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134642176/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698705.043590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698705.043593:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698705.043596:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698705.043602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698705.043620:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698705.043630:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698705/real 1787698705] req@ffff9d85c0fb1f80 x1874531134642304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698705.043640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134642304/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698705.043647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698705.043650:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698705.043653:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642304:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698705.619039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698706.643014:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698707.667026:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698708.691009:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698709.715021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698710.162954:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698710.162962:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698710.162965:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698710.162974:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6611 00000100:00080000:2.0:1787698710.162983:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698710.162989:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698710.163020:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134642432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698710.163038:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698710.163041:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698710.163042:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698710.163045:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6611 00000100:00080000:2.0:1787698710.163046:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698710.163048:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698710.163055:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698710.163065:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134642560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698710.163071:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698710.163073:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698710.163074:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698710.163076:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6611 00000100:00080000:2.0:1787698710.163078:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698710.163083:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698710.163084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134642432:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787698710.163088:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698710.163100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134642688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698710.163115:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698710.163125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698710.163127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698710.163131:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6611 00000100:00080000:2.0:1787698710.163138:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698710.163141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698710.163145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698710.163154:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134642816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698710.163160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698710.163162:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698710.163164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698710.163169:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6611 00000100:00080000:2.0:1787698710.163172:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698710.163176:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134642560:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698710.163181:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698710.163187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698710.163200:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134642944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698710.163210:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698710.163213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698710.163215:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134642688:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698710.163215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698710.163222:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6611 00000100:00080000:2.0:1787698710.163226:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698710.163229:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698710.163233:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698710.163238:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134642816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698710.163244:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134643072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698710.163252:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698710.163254:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642944:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698710.163255:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698710.163258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698710.163270:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6611 00000100:00080000:2.0:1787698710.163273:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698710.163275:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643072:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698710.163279:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698710.163285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698710.163294:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134643200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698710.163312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643200:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698710.739020:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698711.762950:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698712.786998:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698713.235145:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:3.0:1787698713.235186:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698713.235201:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698713.235243:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698713.235252:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698713.235259:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698713.235266:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698713.235273:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698713.235279:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698713.236068:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698710/real 1787698713] req@ffff9d85babd6a00 x1874531134643200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698713.236097:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134643200/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698713.236108:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698713.236112:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698713.236117:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698713.236124:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698710/real 1787698713] req@ffff9d85babd6680 x1874531134643072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698713.236134:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134643072/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698713.236142:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698713.236146:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698713.236150:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698713.236161:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698710/real 1787698713] req@ffff9d85babd4380 x1874531134642688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698713.236173:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134642688/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698713.236180:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698713.236183:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698713.236187:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134642688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698713.236195:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698710/real 1787698713] req@ffff9d85babd6300 x1874531134642816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698713.236205:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134642816/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698713.236226:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698713.236230:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698713.236232:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134642816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698713.236239:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698710/real 1787698713] req@ffff9d85babd7480 x1874531134642944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698713.236256:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134642944/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698713.236263:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698713.236266:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698713.236268:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134642944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698713.236275:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698710/real 1787698713] req@ffff9d85babd5c00 x1874531134642432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698713.236286:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134642432/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698713.236292:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698713.236294:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698713.236297:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134642432:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698713.236302:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698710/real 1787698713] req@ffff9d85babd4000 x1874531134642560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698713.236311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134642560/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698713.236318:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698713.236321:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698713.236323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134642560:192.168.201.137@tcp:38:kworker.0 00010000:00010000:2.0:1787698713.811004:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698714.835020:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698715.282944:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698715.282951:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698715.282953:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698715.282962:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6617 00000100:00080000:2.0:1787698715.282971:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698715.282976:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698715.283002:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134643328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698715.283022:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698715.283024:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698715.283025:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698715.283028:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6617 00000100:00080000:2.0:1787698715.283030:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698715.283032:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698715.283040:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698715.283052:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134643456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698715.283062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698715.283065:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698715.283067:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698715.283070:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134643328:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698715.283072:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6617 00000100:00080000:2.0:1787698715.283079:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698715.283083:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698715.283088:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698715.283100:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134643584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698715.283109:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698715.283113:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698715.283114:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698715.283117:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6617 00000100:00080000:2.0:1787698715.283122:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698715.283125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698715.283130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698715.283133:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698715.283145:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134643712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698715.283153:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698715.283156:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698715.283158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698715.283161:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6617 00000100:00080000:2.0:1787698715.283164:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698715.283167:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698715.283172:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698715.283174:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698715/real 1787698715] req@ffff9d85babd7800 x1874531134643328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698715.283184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134643840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698715.283193:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698715.283196:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134643328/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698715.283198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698715.283202:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698715.283210:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6617 00000100:00080000:1.0:1787698715.283215:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698715.283216:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698715.283222:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698715.283224:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787698715.283228:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698715.283231:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134643328:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698715.283242:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134643968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698715.283256:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698715.283258:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134643456:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698715.283274:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698715.283278:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698715.283283:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698715.283284:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6617 00000100:00080000:2.0:1787698715.283290:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698715.283294:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698715.283300:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698715.283301:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698715/real 1787698715] req@ffff9d85babd6d80 x1874531134643456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698715.283316:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134644096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698715.283318:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134643456/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698715.283355:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698715.283360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698715.283364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134643456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698715.283371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134643584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698715.283389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698715.283400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698715/real 1787698715] req@ffff9d85babd5500 x1874531134643584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698715.283410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134643584/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698715.283417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698715.283419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698715.283423:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134643584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698715.283428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134643712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698715.283443:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698715.283451:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698715/real 1787698715] req@ffff9d85babd4000 x1874531134643712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698715.283462:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134643712/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698715.283468:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698715.283471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698715.283474:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134643712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698715.283479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643840:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698715.283493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698715.283502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698715/real 1787698715] req@ffff9d85babd5c00 x1874531134643840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698715.283511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134643840/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698715.283518:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698715.283521:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698715.283524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698715.283538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698715.283553:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698715.283562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698715/real 1787698715] req@ffff9d85babd7480 x1874531134643968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698715.283578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134643968/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698715.283586:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698715.283588:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698715.283591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134643968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698715.283596:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698715.283609:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698715.283617:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698715/real 1787698715] req@ffff9d85babd6300 x1874531134644096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698715.283627:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134644096/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698715.283634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698715.283637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698715.283640:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644096:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698715.859032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698716.883027:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698717.907034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698718.931095:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698719.955030:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698720.402990:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698720.402999:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698720.403004:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698720.403014:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6622 00000100:00080000:3.0:1787698720.403023:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698720.403031:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698720.403069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134644224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698720.403096:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698720.403100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698720.403102:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698720.403107:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6622 00000100:00080000:3.0:1787698720.403110:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698720.403113:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698720.403122:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698720.403135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134644352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698720.403145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698720.403149:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698720.403151:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698720.403155:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6622 00000100:00080000:3.0:1787698720.403158:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698720.403161:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698720.403165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698720.403171:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134644224:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698720.403181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134644480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698720.403190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698720.403193:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698720.403195:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698720.403199:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6622 00000100:00080000:3.0:1787698720.403202:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698720.403206:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698720.403211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698720.403223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134644608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698720.403232:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698720.403236:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698720.403237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698720.403241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698720.403245:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6622 00000100:00080000:3.0:1787698720.403250:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698720.403254:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698720.403258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698720.403266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134644736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698720.403272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698720.403274:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698720.403275:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698720.403277:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6622 00000100:00080000:3.0:1787698720.403281:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698720.403283:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698720.403287:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698720.403288:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698720/real 1787698720] req@ffff9d85c0fb1c00 x1874531134644224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698720.403295:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134644864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698720.403312:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698720.403315:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698720.403316:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698720.403321:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6622 00000100:00100000:1.0:1787698720.403322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134644224/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698720.403323:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698720.403329:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698720.403337:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698720.403351:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134644992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698720.403352:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698720.403359:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698720.403366:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134644224:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698720.403377:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134644352:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698720.403400:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698720.403412:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698720/real 1787698720] req@ffff9d85c0fb0a80 x1874531134644352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698720.403425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134644352/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698720.403432:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698720.403435:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698720.403438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134644352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698720.403460:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134644480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698720.403479:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698720.403490:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698720/real 1787698720] req@ffff9d85c0fb3b80 x1874531134644480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698720.403501:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134644480/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698720.403508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698720.403517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698720.403520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134644480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698720.403527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134644608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698720.403545:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698720.403555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698720/real 1787698720] req@ffff9d85c0fb0380 x1874531134644608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698720.403563:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134644608/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698720.403575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698720.403583:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698720.403587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134644608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698720.403592:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644736:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698720.403612:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698720.403623:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698720/real 1787698720] req@ffff9d85c0fb2d80 x1874531134644736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698720.403630:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134644736/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698720.403635:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698720.403637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698720.403639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698720.403642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698720.403653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698720.403660:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698720/real 1787698720] req@ffff9d85c0fb3800 x1874531134644864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698720.403685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134644864/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698720.403693:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698720.403696:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698720.403699:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698720.403711:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698720.403733:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698720.403744:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698720/real 1787698720] req@ffff9d85c0fb0000 x1874531134644992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698720.403755:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134644992/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698720.403763:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698720.403765:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698720.403768:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134644992:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698720.979084:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698722.002979:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698723.027110:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698724.051023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698725.075001:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698725.522993:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698725.523004:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698725.523007:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698725.523017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6627 00000100:00080000:2.0:1787698725.523028:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698725.523034:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698725.523078:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134645120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698725.523109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698725.523113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698725.523115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698725.523119:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6627 00000100:00080000:2.0:1787698725.523124:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698725.523136:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698725.523141:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134645120:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787698725.523148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698725.523165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134645248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698725.523177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698725.523180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698725.523182:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698725.523186:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6627 00000100:00080000:2.0:1787698725.523190:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698725.523194:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698725.523198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698725.523205:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698725.523226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134645376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698725.523229:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698725/real 1787698725] req@ffff9d85babd4380 x1874531134645120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698725.523251:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698725.523255:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698725.523258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698725.523259:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134645120/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698725.523273:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6627 00000100:00080000:2.0:1787698725.523280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698725.523282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698725.523287:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698725.523291:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:2.0:1787698725.523295:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698725.523299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134645120:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698725.523310:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134645504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698725.523323:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698725.523326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134645248:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698725.523330:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698725.523333:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698725.523338:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6627 00000100:00080000:2.0:1787698725.523342:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698725.523347:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698725.523351:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698725.523355:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698725.523362:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698725/real 1787698725] req@ffff9d85babd6680 x1874531134645248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698725.523370:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134645632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698725.523374:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134645248/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698725.523395:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698725.523397:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698725.523401:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698725.523405:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698725.523406:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698725.523412:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134645248:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698725.523414:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6627 00000100:00080000:2.0:1787698725.523421:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698725.523425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134645376:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698725.523429:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698725.523437:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698725.523447:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698725.523453:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134645760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698725.523458:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698725/real 1787698725] req@ffff9d85babd6a00 x1874531134645376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698725.523478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134645376/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698725.523487:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698725.523494:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698725.523495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698725.523496:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698725.523503:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698725.523504:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6627 00000100:00080000:2.0:1787698725.523510:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698725.523511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134645376:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698725.523517:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698725.523524:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698725.523534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134645504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787698725.523538:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134645888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698725.523561:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698725.523574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698725/real 1787698725] req@ffff9d85babd5180 x1874531134645504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698725.523613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134645504/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698725.523619:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698725.523621:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698725.523623:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134645504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698725.523627:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134645632:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698725.523641:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698725.523654:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698725/real 1787698725] req@ffff9d85babd7100 x1874531134645632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698725.523661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134645632/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698725.523665:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698725.523667:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698725.523669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134645632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698725.523672:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134645760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698725.523682:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698725.523687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698725/real 1787698725] req@ffff9d85babd4a80 x1874531134645760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698725.523694:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134645760/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698725.523698:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698725.523699:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698725.523701:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134645760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698725.523724:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134645888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698725.523742:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698725.523752:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698725/real 1787698725] req@ffff9d85babd7b80 x1874531134645888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698725.523767:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134645888/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698725.523775:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698725.523778:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698725.523781:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134645888:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698726.099100:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698727.123038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698728.147007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698729.170947:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698730.195029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698730.642963:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698730.642970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698730.642973:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698730.642980:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6632 00000100:00080000:3.0:1787698730.642989:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698730.642994:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698730.643028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134646016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698730.643052:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698730.643055:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698730.643057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698730.643060:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6632 00000100:00080000:3.0:1787698730.643063:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698730.643067:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698730.643076:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698730.643088:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134646016:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698730.643093:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134646144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698730.643106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698730.643109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698730.643111:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698730.643114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6632 00000100:00080000:3.0:1787698730.643117:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698730.643120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698730.643124:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698730.643132:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134646272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698730.643137:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698730.643140:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698730.643141:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698730.643143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6632 00000100:00080000:3.0:1787698730.643144:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698730.643146:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698730.643149:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698730.643155:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134646400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698730.643161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698730.643163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698730.643164:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698730.643166:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6632 00000100:00080000:3.0:1787698730.643169:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698730.643171:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698730.643173:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698730.643179:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698730.643181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134646528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698730.643187:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698730.643190:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698730.643191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698730.643200:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6632 00000100:00080000:3.0:1787698730.643203:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698730.643205:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698730.643211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698730.643220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134646656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698730.643228:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698730/real 1787698730] req@ffff9d85c0fb0e00 x1874531134646016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698730.643232:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698730.643244:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698730.643247:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698730.643252:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134646016/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698730.643255:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6632 00000100:00080000:3.0:1787698730.643264:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698730.643271:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698730.643275:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698730.643279:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787698730.643279:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698730.643285:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134646016:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698730.643297:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134646784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698730.643305:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134646144:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698730.643330:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698730.643353:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698730/real 1787698730] req@ffff9d85c0fb2680 x1874531134646144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698730.643365:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134646144/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698730.643372:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698730.643376:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698730.643378:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134646144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698730.643384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134646272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698730.643402:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698730.643412:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698730/real 1787698730] req@ffff9d85c0fb2a00 x1874531134646272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698730.643422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134646272/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698730.643429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698730.643432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698730.643436:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134646272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698730.643443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134646400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698730.643460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698730.643470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698730/real 1787698730] req@ffff9d85c0fb0000 x1874531134646400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698730.643481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134646400/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698730.643489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698730.643493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698730.643507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134646400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698730.643511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134646528:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698730.643525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698730.643533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698730/real 1787698730] req@ffff9d85c0fb3800 x1874531134646528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698730.643540:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134646528/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698730.643545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698730.643547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698730.643548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134646528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698730.643551:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134646656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698730.643560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698730.643565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698730/real 1787698730] req@ffff9d85c0fb2d80 x1874531134646656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698730.643572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134646656/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698730.643576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698730.643578:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698730.643579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134646656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698730.643593:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134646784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698730.643610:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698730.643621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698730/real 1787698730] req@ffff9d85c0fb0380 x1874531134646784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698730.643631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134646784/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698730.643638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698730.643641:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698730.643644:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134646784:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698731.219066:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698732.243012:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698733.267011:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698734.291022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698735.314986:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698735.762935:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698735.762945:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698735.762949:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698735.762959:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6637 00000100:00080000:3.0:1787698735.762970:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698735.762976:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698735.763011:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134646912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698735.763035:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698735.763039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698735.763041:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698735.763045:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6637 00000100:00080000:3.0:1787698735.763048:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698735.763052:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698735.763060:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698735.763072:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134647040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698735.763079:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134646912:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698735.763086:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698735.763090:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698735.763092:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698735.763096:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6637 00000100:00080000:3.0:1787698735.763100:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698735.763104:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698735.763109:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698735.763120:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134647168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698735.763128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698735.763132:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698735.763136:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698735.763137:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698735.763143:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6637 00000100:00080000:3.0:1787698735.763147:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698735.763152:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698735.763157:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698735.763168:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134647296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698735.763177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698735.763181:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698735/real 1787698735] req@ffff9d85c0fb3b80 x1874531134646912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698735.763182:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698735.763187:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698735.763195:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6637 00000100:00080000:3.0:1787698735.763199:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698735.763201:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134646912/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698735.763208:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698735.763216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698735.763219:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698735.763224:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698735.763231:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134646912:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698735.763231:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134647424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698735.763243:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698735.763246:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698735.763247:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698735.763252:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647040:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698735.763255:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6637 00000100:00080000:3.0:1787698735.763261:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698735.763265:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698735.763270:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787698735.763270:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698735.763281:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698735/real 1787698735] req@ffff9d85c0fb0a80 x1874531134647040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698735.763286:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134647552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698735.763295:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134647040/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698735.763301:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698735.763304:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698735.763307:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698735.763308:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698735.763312:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698735.763317:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6637 00000100:00100000:1.0:1787698735.763321:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647040:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698735.763321:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698735.763328:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698735.763335:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647168:192.168.201.137@tcp:8:kworker.0 00000008:00100000:3.0:1787698735.763335:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698735.763352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134647680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698735.763356:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698735.763365:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698735/real 1787698735] req@ffff9d85c0fb1c00 x1874531134647168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698735.763389:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134647168/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698735.763397:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698735.763401:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698735.763405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698735.763437:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698735.763458:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698735.763469:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698735/real 1787698735] req@ffff9d85c0fb1f80 x1874531134647296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698735.763478:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134647296/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698735.763484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698735.763486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698735.763488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698735.763491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134647424:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698735.763504:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698735.763509:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698735/real 1787698735] req@ffff9d85c0fb3100 x1874531134647424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698735.763516:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134647424/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698735.763520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698735.763522:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698735.763523:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134647424:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698735.763533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134647552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698735.763543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698735.763548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698735/real 1787698735] req@ffff9d85c0fb1880 x1874531134647552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698735.763554:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134647552/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698735.763559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698735.763560:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698735.763562:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134647552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698735.763565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134647680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698735.763572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698735.763577:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698735/real 1787698735] req@ffff9d85c0fb1500 x1874531134647680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698735.763583:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134647680/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698735.763588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698735.763589:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698735.763591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134647680:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698736.339013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698737.362995:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698738.387013:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698739.410983:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698740.434996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698740.882965:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698740.882975:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698740.882980:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698740.882987:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6642 00000100:00080000:0.0:1787698740.882997:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698740.883002:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698740.883031:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134647808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698740.883051:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698740.883053:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698740.883054:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698740.883057:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6642 00000100:00080000:0.0:1787698740.883059:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698740.883061:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698740.883068:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698740.883075:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134647936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698740.883082:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698740.883086:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698740.883087:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134647808:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698740.883088:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698740.883094:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6642 00000100:00080000:0.0:1787698740.883096:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698740.883099:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698740.883102:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698740.883111:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134648064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698740.883117:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698740.883119:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698740.883120:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698740.883122:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6642 00000100:00080000:0.0:1787698740.883131:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698740.883133:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698740.883135:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698740.883139:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698740.883143:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134648192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698740.883150:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698740.883152:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698740.883154:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698740.883156:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6642 00000100:00080000:0.0:1787698740.883158:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698740.883166:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698740.883169:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698740.883174:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698740/real 1787698740] req@ffff9d85b9d0b480 x1874531134647808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698740.883177:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134648320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698740.883194:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698740.883198:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134647808/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698740.883210:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698740.883212:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698740.883215:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134647808:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698740.883222:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698740.883224:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698740.883227:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6642 00000100:00080000:0.0:1787698740.883230:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698740.883232:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698740.883236:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698740.883238:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698740.883254:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134648448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698740.883259:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698740.883268:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698740.883271:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698740.883273:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698740.883273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698740/real 1787698740] req@ffff9d85b9d0a680 x1874531134647936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698740.883280:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6642 00000100:00080000:0.0:1787698740.883285:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698740.883289:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698740.883292:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134647936/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698740.883293:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698740.883304:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134648576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698740.883310:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698740.883317:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698740.883330:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134647936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698740.883335:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698740.883352:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698740.883359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698740/real 1787698740] req@ffff9d85b9d0bb80 x1874531134648064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698740.883367:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134648064/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698740.883371:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698740.883373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698740.883375:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698740.883378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698740.883387:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698740.883392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698740/real 1787698740] req@ffff9d85b9d0ad80 x1874531134648192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698740.883398:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134648192/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698740.883403:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698740.883404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698740.883406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698740.883409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134648320:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698740.883417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698740.883425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698740/real 1787698740] req@ffff9d85b9d08000 x1874531134648320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698740.883435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134648320/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698740.883441:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698740.883444:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698740.883446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134648320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698740.883461:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134648448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698740.883477:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698740.883487:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698740/real 1787698740] req@ffff9d85b9d08a80 x1874531134648448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698740.883498:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134648448/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698740.883504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698740.883507:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698740.883510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134648448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698740.883514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134648576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698740.883528:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698740.883536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698740/real 1787698740] req@ffff9d85b9d08e00 x1874531134648576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698740.883545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134648576/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698740.883551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698740.883553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698740.883556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134648576:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698741.458993:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698742.482988:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698743.506989:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698744.530990:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698745.555033:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698746.002992:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698746.003007:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698746.003010:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698746.003021:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6647 00000100:00080000:2.0:1787698746.003030:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698746.003035:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698746.003068:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134648704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698746.003089:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698746.003093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698746.003095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698746.003099:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6647 00000100:00080000:2.0:1787698746.003103:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698746.003106:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698746.003116:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698746.003131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134648832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698746.003134:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134648704:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698746.003151:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698746.003154:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698746.003156:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698746.003160:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6647 00000100:00080000:2.0:1787698746.003163:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698746.003166:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698746.003171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698746.003180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134648960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698746.003187:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698746.003190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698746.003191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698746.003195:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6647 00000800:00000100:1.0:1787698746.003196:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698746.003199:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698746.003203:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698746.003206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698746.003215:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134649088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698746.003221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698746.003224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698746.003225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698746.003228:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6647 00000100:00080000:2.0:1787698746.003230:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698746.003246:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698746.003250:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698746.003256:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698746/real 1787698746] req@ffff9d85babd5f80 x1874531134648704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698746.003261:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134649216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698746.003280:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698746.003283:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698746.003286:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134648704/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698746.003286:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698746.003292:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6647 00000100:00080000:2.0:1787698746.003299:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698746.003305:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698746.003312:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698746.003317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698746.003325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698746.003330:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134648704:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698746.003330:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134649344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698746.003344:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698746.003346:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698746.003347:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698746.003350:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6647 00000100:00080000:2.0:1787698746.003352:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698746.003355:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698746.003358:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698746.003359:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698746.003370:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134649472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698746.003386:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698746.003398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698746/real 1787698746] req@ffff9d85babd4700 x1874531134648832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698746.003407:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134648832/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698746.003412:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698746.003415:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698746.003417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698746.003421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698746.003431:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698746.003437:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698746/real 1787698746] req@ffff9d85babd4e00 x1874531134648960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698746.003444:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134648960/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698746.003450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698746.003452:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698746.003454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134648960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698746.003464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698746.003475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698746.003481:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698746/real 1787698746] req@ffff9d85babd7b80 x1874531134649088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698746.003488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134649088/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698746.003494:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698746.003495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698746.003497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698746.003500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134649216:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698746.003510:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698746.003516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698746/real 1787698746] req@ffff9d85babd4a80 x1874531134649216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698746.003523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134649216/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698746.003528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698746.003529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698746.003531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134649216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698746.003534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134649344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698746.003543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698746.003549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698746/real 1787698746] req@ffff9d85babd7100 x1874531134649344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698746.003556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134649344/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698746.003561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698746.003563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698746.003565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134649344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698746.003575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134649472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698746.003585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698746.003591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698746/real 1787698746] req@ffff9d85babd5180 x1874531134649472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698746.003597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134649472/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698746.003602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698746.003604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698746.003605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134649472:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698746.579023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698747.603039:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698748.626986:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698749.651018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698750.675026:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698751.122974:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698751.122981:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698751.122985:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698751.122992:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6652 00000100:00080000:0.0:1787698751.123003:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698751.123011:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698751.123051:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134649600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698751.123080:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698751.123084:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698751.123085:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698751.123089:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6652 00000100:00080000:0.0:1787698751.123092:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698751.123096:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698751.123103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698751.123115:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134649728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698751.123123:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698751.123125:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698751.123127:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698751.123129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6652 00000100:00080000:0.0:1787698751.123132:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698751.123135:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698751.123138:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698751.123158:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134649600:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698751.123165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134649856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698751.123178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698751.123182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698751.123184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698751.123189:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6652 00000100:00080000:0.0:1787698751.123194:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698751.123199:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698751.123205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698751.123216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134649984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698751.123223:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698751.123226:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698751.123227:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698751.123230:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6652 00000100:00080000:0.0:1787698751.123233:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698751.123238:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698751.123239:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698751.123246:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698751.123254:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134650112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698751.123262:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698751.123264:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698751.123266:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698751.123268:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6652 00000100:00080000:0.0:1787698751.123270:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698751.123272:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698751.123276:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698751.123284:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134650240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698751.123291:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698751/real 1787698751] req@ffff9d85b9d08e00 x1874531134649600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.123300:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698751.123305:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698751.123308:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698751.123315:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6652 00000100:00080000:0.0:1787698751.123317:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698751.123320:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698751.123323:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698751.123328:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134649600/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.123333:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134650368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698751.123365:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698751.123369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698751.123373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134649600:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698751.123380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649728:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698751.123400:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698751.123409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698751/real 1787698751] req@ffff9d85b9d08a80 x1874531134649728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698751.123417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134649728/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698751.123423:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698751.123425:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698751.123427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698751.123445:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698751.123456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698751.123463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698751/real 1787698751] req@ffff9d85b9d08000 x1874531134649856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698751.123470:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134649856/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698751.123475:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698751.123476:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698751.123478:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698751.123482:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698751.123492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698751.123497:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698751/real 1787698751] req@ffff9d85b9d0ad80 x1874531134649984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698751.123504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134649984/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698751.123509:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698751.123511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698751.123513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134649984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698751.123516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134650112:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698751.123525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698751.123531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698751/real 1787698751] req@ffff9d85b9d0bb80 x1874531134650112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698751.123538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134650112/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698751.123543:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698751.123544:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698751.123546:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134650112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698751.123549:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134650240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698751.123559:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698751.123564:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698751/real 1787698751] req@ffff9d85b9d0a680 x1874531134650240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698751.123572:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134650240/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698751.123577:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698751.123579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698751.123580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134650240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698751.123583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134650368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698751.123592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698751.123597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698751/real 1787698751] req@ffff9d85b9d0b480 x1874531134650368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698751.123604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134650368/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698751.123609:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698751.123610:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698751.123612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134650368:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698751.699035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698752.723033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698753.747022:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698754.771064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698755.795085:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698756.243039:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698756.243051:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698756.243056:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698756.243086:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6658 00000100:00080000:3.0:1787698756.243096:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698756.243102:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698756.243139:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134650496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698756.243174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698756.243177:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698756.243179:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698756.243183:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6658 00000100:00080000:3.0:1787698756.243187:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698756.243190:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698756.243197:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698756.243209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134650624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698756.243218:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698756.243221:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134650496:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698756.243221:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698756.243226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698756.243232:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6658 00000100:00080000:3.0:1787698756.243234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698756.243236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698756.243240:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698756.243249:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134650752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698756.243256:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698756.243258:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698756.243260:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698756.243262:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6658 00000100:00080000:3.0:1787698756.243265:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698756.243266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698756.243269:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698756.243278:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134650880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698756.243282:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698756.243290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698756.243293:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698756.243295:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698756.243299:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6658 00000100:00080000:3.0:1787698756.243301:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698756.243304:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698756.243307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698756.243314:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134651008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698756.243320:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698756.243322:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698756.243323:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698756.243328:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6658 00000100:00100000:1.0:1787698756.243329:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698756/real 1787698756] req@ffff9d85c0fb2300 x1874531134650496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698756.243330:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698756.243337:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698756.243345:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698756.243357:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134650496/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698756.243358:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134651136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698756.243382:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698756.243389:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698756.243390:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698756.243391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698756.243395:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6658 00000100:00080000:1.0:1787698756.243399:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698756.243400:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698756.243406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134650496:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698756.243406:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698756.243415:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698756.243422:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134651264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698756.243439:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134650624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698756.243457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698756.243466:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698756/real 1787698756] req@ffff9d85c0fb3480 x1874531134650624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698756.243486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134650624/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698756.243495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698756.243499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698756.243502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134650624:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698756.243509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134650752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698756.243538:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698756.243547:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698756/real 1787698756] req@ffff9d85c0fb0700 x1874531134650752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698756.243555:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134650752/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698756.243561:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698756.243563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698756.243565:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134650752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698756.243570:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134650880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698756.243582:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698756.243588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698756/real 1787698756] req@ffff9d85c0fb1500 x1874531134650880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698756.243595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134650880/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698756.243600:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698756.243602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698756.243604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134650880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698756.243607:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651008:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698756.243618:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698756.243624:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698756/real 1787698756] req@ffff9d85c0fb1880 x1874531134651008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698756.243631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134651008/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698756.243636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698756.243638:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698756.243639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698756.243642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698756.243651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698756.243657:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698756/real 1787698756] req@ffff9d85c0fb3100 x1874531134651136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698756.243664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134651136/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698756.243669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698756.243670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698756.243672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698756.243675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698756.243684:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698756.243689:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698756/real 1787698756] req@ffff9d85c0fb1f80 x1874531134651264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698756.243695:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134651264/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698756.243700:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698756.243701:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698756.243703:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651264:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698756.819126:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698757.843109:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698758.867041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698759.891021:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698760.914996:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698761.362944:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698761.362954:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698761.362958:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698761.362969:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6663 00000100:00080000:2.0:1787698761.363009:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698761.363017:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698761.363053:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134651392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698761.363081:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698761.363085:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698761.363087:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698761.363091:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6663 00000100:00080000:2.0:1787698761.363096:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698761.363101:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698761.363110:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698761.363123:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134651392:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698761.363128:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134651520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698761.363140:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698761.363144:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698761.363147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698761.363151:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6663 00000100:00080000:2.0:1787698761.363156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698761.363160:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698761.363166:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698761.363177:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134651648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698761.363186:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698761.363195:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698761.363199:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698761.363202:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698761.363207:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6663 00000100:00080000:2.0:1787698761.363211:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698761.363216:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698761.363220:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698761.363229:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698761/real 1787698761] req@ffff9d85babd6a00 x1874531134651392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698761.363234:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134651776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698761.363261:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698761.363264:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698761.363267:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698761.363269:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134651392/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698761.363275:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6663 00000100:00080000:2.0:1787698761.363283:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698761.363291:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698761.363300:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698761.363301:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698761.363307:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698761.363311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134651392:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698761.363317:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134651904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698761.363329:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698761.363333:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698761.363336:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134651520:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698761.363338:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698761.363345:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6663 00000100:00080000:2.0:1787698761.363351:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698761.363356:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698761.363361:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698761.363362:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698761.363374:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698761/real 1787698761] req@ffff9d85babd6680 x1874531134651520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698761.363381:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134652032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698761.363390:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134651520/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698761.363403:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698761.363404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698761.363410:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698761.363414:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134651520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698761.363420:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134651648:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698761.363420:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698761.363425:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698761.363435:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6663 00000100:00080000:2.0:1787698761.363440:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698761.363445:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698761.363452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698761.363454:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698761.363469:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698761/real 1787698761] req@ffff9d85babd4380 x1874531134651648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698761.363472:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134652160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698761.363499:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134651648/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698761.363507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698761.363510:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698761.363512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134651648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698761.363519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134651776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698761.363540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698761.363551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698761/real 1787698761] req@ffff9d85babd6300 x1874531134651776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698761.363564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134651776/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698761.363573:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698761.363577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698761.363580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134651776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698761.363586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651904:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698761.363602:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698761.363609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698761/real 1787698761] req@ffff9d85babd7480 x1874531134651904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698761.363617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134651904/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698761.363622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698761.363624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698761.363626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134651904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698761.363643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698761.363662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698761.363673:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698761/real 1787698761] req@ffff9d85babd5c00 x1874531134652032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698761.363686:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134652032/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698761.363692:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698761.363694:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698761.363696:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698761.363700:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698761.363714:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698761.363721:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698761/real 1787698761] req@ffff9d85babd4000 x1874531134652160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698761.363728:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134652160/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698761.363734:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698761.363736:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698761.363738:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652160:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698761.939035:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698762.963029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698763.987030:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698765.010987:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698766.034998:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698766.482948:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698766.482956:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698766.482959:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698766.482967:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6668 00000100:00080000:2.0:1787698766.482976:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698766.482982:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698766.483014:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134652288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698766.483038:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698766.483041:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698766.483042:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698766.483045:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6668 00000100:00080000:2.0:1787698766.483048:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698766.483050:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698766.483057:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698766.483066:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134652416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698766.483073:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698766.483076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698766.483077:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698766.483081:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6668 00000100:00080000:2.0:1787698766.483083:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698766.483085:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134652288:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698766.483086:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698766.483094:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698766.483101:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134652544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698766.483107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698766.483110:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698766.483111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698766.483114:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6668 00000100:00080000:2.0:1787698766.483116:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698766.483119:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698766.483122:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698766.483129:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134652672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698766.483134:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698766.483136:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698766.483137:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698766.483140:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6668 00000100:00080000:2.0:1787698766.483142:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698766.483146:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698766.483148:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698766.483157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134652800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698766.483158:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698766.483167:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698766.483170:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698766.483171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698766.483175:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6668 00000100:00080000:2.0:1787698766.483177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698766.483180:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698766.483183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698766.483191:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134652928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698766.483197:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698766.483198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698766.483199:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698766.483202:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6668 00000100:00080000:2.0:1787698766.483204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698766.483207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698766.483210:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698766.483215:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698766/real 1787698766] req@ffff9d85babd5500 x1874531134652288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698766.483220:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134653056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698766.483244:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134652288/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698766.483258:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698766.483261:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698766.483264:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134652288:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698766.483282:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134652416:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698766.483299:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698766.483307:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698766/real 1787698766] req@ffff9d85babd6d80 x1874531134652416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698766.483315:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134652416/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698766.483321:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698766.483323:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698766.483325:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134652416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698766.483329:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134652544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698766.483339:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698766.483345:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698766/real 1787698766] req@ffff9d85babd7800 x1874531134652544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698766.483352:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134652544/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698766.483357:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698766.483358:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698766.483368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134652544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698766.483372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134652672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698766.483384:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698766.483390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698766/real 1787698766] req@ffff9d85babd4000 x1874531134652672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698766.483397:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134652672/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698766.483403:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698766.483404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698766.483406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134652672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698766.483409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652800:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698766.483419:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698766.483425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698766/real 1787698766] req@ffff9d85babd5c00 x1874531134652800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698766.483431:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134652800/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698766.483436:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698766.483438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698766.483440:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698766.483443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698766.483452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698766.483463:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698766/real 1787698766] req@ffff9d85babd7480 x1874531134652928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698766.483472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134652928/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698766.483477:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698766.483480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698766.483482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134652928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698766.483485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653056:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698766.483495:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698766.483500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698766/real 1787698766] req@ffff9d85babd6300 x1874531134653056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698766.483508:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134653056/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698766.483513:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698766.483515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698766.483516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653056:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698767.059041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698768.083000:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698769.107034:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698770.130978:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698771.155017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698771.602997:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698771.603006:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698771.603009:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698771.603017:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6673 00000100:00080000:0.0:1787698771.603026:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698771.603032:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698771.603065:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134653184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698771.603089:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698771.603092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698771.603094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698771.603097:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6673 00000100:00080000:0.0:1787698771.603099:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698771.603101:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698771.603109:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698771.603118:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134653312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698771.603127:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698771.603130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698771.603131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698771.603137:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134653184:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698771.603140:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6673 00000100:00080000:0.0:1787698771.603148:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698771.603152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698771.603158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698771.603182:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134653440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698771.603192:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698771.603194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698771.603196:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698771.603198:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6673 00000100:00080000:0.0:1787698771.603202:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698771.603207:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698771.603209:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698771.603215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698771.603231:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134653568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698771.603239:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698771.603241:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698771.603243:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698771.603247:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698771/real 1787698771] req@ffff9d85b9d09c00 x1874531134653184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698771.603249:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6673 00000100:00080000:0.0:1787698771.603254:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698771.603260:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698771.603266:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698771.603276:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134653696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698771.603283:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134653184/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698771.603285:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698771.603293:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698771.603298:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698771.603306:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6673 00000100:00080000:0.0:1787698771.603314:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698771.603319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698771.603321:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698771.603325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:0.0:1787698771.603331:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698771.603331:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134653184:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698771.603352:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134653824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698771.603359:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134653312:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698771.603372:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698771.603377:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698771.603379:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698771.603384:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6673 00000100:00080000:0.0:1787698771.603388:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698771.603392:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698771.603395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698771.603401:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698771.603410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698771/real 1787698771] req@ffff9d85b9d08700 x1874531134653312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698771.603419:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134653952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698771.603424:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134653312/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698771.603446:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698771.603449:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698771.603452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134653312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698771.603457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134653440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698771.603471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698771.603480:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698771/real 1787698771] req@ffff9d85b9d09500 x1874531134653440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698771.603488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134653440/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698771.603493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698771.603495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698771.603497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134653440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698771.603502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134653568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698771.603512:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698771.603517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698771/real 1787698771] req@ffff9d85b9d09180 x1874531134653568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698771.603525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134653568/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698771.603530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698771.603532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698771.603533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134653568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698771.603537:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653696:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698771.603546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698771.603552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698771/real 1787698771] req@ffff9d85b9d0a300 x1874531134653696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698771.603559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134653696/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698771.603564:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698771.603565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698771.603567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698771.603577:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698771.603587:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698771.603593:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698771/real 1787698771] req@ffff9d85b9d09f80 x1874531134653824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698771.603600:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134653824/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698771.603606:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698771.603607:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698771.603609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698771.603613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698771.603629:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698771.603639:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698771/real 1787698771] req@ffff9d85b9d0b800 x1874531134653952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698771.603653:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134653952/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698771.603662:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698771.603665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698771.603669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134653952:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698772.179058:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698773.203010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698774.227008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698775.251062:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698776.275038:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698776.722999:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698776.723013:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698776.723018:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698776.723031:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6678 00000100:00080000:1.0:1787698776.723046:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698776.723055:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698776.723105:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134654080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698776.723134:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698776.723138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698776.723140:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698776.723145:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6678 00000100:00080000:1.0:1787698776.723150:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698776.723155:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698776.723166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698776.723172:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134654080:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698776.723183:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134654208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698776.723194:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698776.723197:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698776.723198:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698776.723201:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6678 00000100:00080000:1.0:1787698776.723204:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698776.723207:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698776.723211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698776.723220:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134654336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698776.723225:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698776.723228:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698776.723229:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698776.723231:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6678 00000100:00080000:1.0:1787698776.723234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698776.723236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698776.723239:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698776.723246:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134654464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698776.723255:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698776.723258:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698776.723260:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698776.723264:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6678 00000100:00080000:1.0:1787698776.723267:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698776.723271:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698776.723276:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698776.723283:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134654208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698776.723288:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134654592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698776.723298:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698776.723301:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698776.723303:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698776.723306:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6678 00000100:00080000:1.0:1787698776.723310:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698776.723314:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698776.723318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134654336:192.168.201.137@tcp:8:kworker.0 00000008:00100000:1.0:1787698776.723320:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698776.723335:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134654720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698776.723351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134654464:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787698776.723363:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698776.723366:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698776.723368:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698776.723372:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6678 00000100:00080000:1.0:1787698776.723376:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698776.723380:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698776.723383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134654592:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787698776.723386:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698776.723401:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134654848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698776.723417:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134654720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698776.723464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134654848:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698777.299103:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698778.322989:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698779.347075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698779.795209:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698779.795267:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698779.795287:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698779.795339:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698779.795357:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698779.795368:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698779.795381:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698779.795393:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698779.795404:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698776/real 1787698779] req@ffff9d85b9d2fb80 x1874531134654208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698779.795404:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698779.795442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134654208/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698779.795457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698779.795463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698779.795468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134654208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698779.795478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698776/real 1787698779] req@ffff9d85b9d2f800 x1874531134654336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698779.795491:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134654336/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698779.795501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698779.795505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698779.795509:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134654336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698779.795516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698776/real 1787698779] req@ffff9d85b9d2ca80 x1874531134654464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698779.795529:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134654464/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698779.795538:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698779.795542:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698779.795545:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134654464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698779.795552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698776/real 1787698779] req@ffff9d85b9d2f480 x1874531134654080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698779.795565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134654080/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698779.795574:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698779.795577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698779.795581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134654080:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698779.795625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698776/real 1787698779] req@ffff9d85b9d2e300 x1874531134654848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698779.795638:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134654848/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698779.795648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698779.795653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698779.795656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134654848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698779.795691:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698776/real 1787698779] req@ffff9d85b9d2c000 x1874531134654592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698779.795705:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134654592/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698779.795714:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698779.795717:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698779.795721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134654592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698779.795728:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698776/real 1787698779] req@ffff9d85b9d2ce00 x1874531134654720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698779.795739:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134654720/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698779.795748:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698779.795751:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698779.795755:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134654720:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698780.370990:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698781.395022:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698781.842959:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698781.842970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698781.842973:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698781.842984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6683 00000100:00080000:1.0:1787698781.842994:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698781.843001:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698781.843033:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134654976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698781.843065:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698781.843068:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698781.843070:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698781.843074:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6683 00000100:00080000:1.0:1787698781.843078:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698781.843082:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698781.843093:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698781.843107:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134655104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698781.843114:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134654976:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698781.843121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698781.843125:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698781.843127:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698781.843132:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6683 00000100:00080000:1.0:1787698781.843136:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698781.843140:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698781.843145:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698781.843157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134655232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698781.843166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698781.843170:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698781.843172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698781.843175:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6683 00000100:00080000:1.0:1787698781.843179:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698781.843183:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698781.843187:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698781.843193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698781.843206:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134655360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698781.843216:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698781.843219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698781.843222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698781.843226:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698781/real 1787698781] req@ffff9d85b9d2dc00 x1874531134654976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698781.843228:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6683 00000100:00080000:1.0:1787698781.843234:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698781.843242:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698781.843248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698781.843255:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134654976/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698781.843261:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134655488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698781.843274:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698781.843277:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698781.843282:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698781.843282:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698781.843285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698781.843290:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134654976:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698781.843294:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6683 00000100:00080000:1.0:1787698781.843300:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698781.843305:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698781.843309:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698781.843321:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134655616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698781.843322:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134655104:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698781.843338:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698781.843342:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698781.843345:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698781.843349:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6683 00000800:00000100:0.0:1787698781.843354:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698781.843354:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698781.843367:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698781.843371:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698781/real 1787698781] req@ffff9d85b9d2e680 x1874531134655104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698781.843374:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698781.843394:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134655104/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698781.843395:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134655744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698781.843406:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698781.843413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698781.843419:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134655104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698781.843426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134655232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698781.843451:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698781.843461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698781/real 1787698781] req@ffff9d85b9d2d500 x1874531134655232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698781.843472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134655232/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698781.843485:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698781.843488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698781.843491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134655232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698781.843497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134655360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698781.843517:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698781.843528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698781/real 1787698781] req@ffff9d85b9d2ed80 x1874531134655360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698781.843538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134655360/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698781.843545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698781.843548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698781.843551:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134655360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698781.843556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134655488:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698781.843574:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698781.843585:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698781/real 1787698781] req@ffff9d85b9d2c700 x1874531134655488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698781.843595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134655488/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698781.843602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698781.843604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698781.843607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134655488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698781.843635:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134655616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698781.843654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698781.843664:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698781/real 1787698781] req@ffff9d85b9d2c380 x1874531134655616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698781.843673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134655616/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698781.843680:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698781.843682:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698781.843685:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134655616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698781.843689:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134655744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698781.843705:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698781.843715:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698781/real 1787698781] req@ffff9d85b9d2dc00 x1874531134655744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698781.843725:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134655744/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698781.843732:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698781.843735:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698781.843738:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134655744:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698782.419022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698783.443046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698784.467049:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698785.491079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698786.515099:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698786.962956:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698786.962966:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698786.962970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698786.962980:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6688 00000100:00080000:1.0:1787698786.962992:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698786.963000:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698786.963042:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134655872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698786.963069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698786.963071:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698786.963072:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698786.963076:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6688 00000100:00080000:1.0:1787698786.963078:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698786.963080:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698786.963088:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698786.963098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134656000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698786.963104:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698786.963106:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698786.963108:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698786.963111:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6688 00000100:00080000:1.0:1787698786.963112:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698786.963115:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698786.963118:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698786.963124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134656128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698786.963129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698786.963133:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698786.963134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698786.963136:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134655872:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698786.963138:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6688 00000100:00080000:1.0:1787698786.963142:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698786.963147:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698786.963150:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698786.963157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134656256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698786.963162:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698786.963165:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698786.963166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698786.963167:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6688 00000100:00080000:1.0:1787698786.963169:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698786.963171:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698786.963174:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698786.963181:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134656384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698786.963186:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698786.963189:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698786.963190:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698786.963193:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6688 00000100:00080000:1.0:1787698786.963204:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698786.963207:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698786.963209:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698786.963217:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134656512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698786.963219:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698786.963226:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698786.963229:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698786.963230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698786.963232:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6688 00000100:00080000:1.0:1787698786.963234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698786.963236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698786.963239:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698786.963254:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134656640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698786.963277:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698786/real 1787698786] req@ffff9d85b9d2ce00 x1874531134655872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698786.963296:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134655872/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698786.963307:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698786.963311:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698786.963315:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134655872:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698786.963323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656000:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698786.963342:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698786.963353:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698786/real 1787698786] req@ffff9d85b9d2c000 x1874531134656000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698786.963364:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134656000/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698786.963372:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698786.963375:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698786.963378:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698786.963408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698786.963428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698786.963438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698786/real 1787698786] req@ffff9d85b9d2e300 x1874531134656128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698786.963450:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134656128/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698786.963464:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698786.963467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698786.963470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698786.963476:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698786.963491:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698786.963501:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698786/real 1787698786] req@ffff9d85b9d2f480 x1874531134656256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698786.963511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134656256/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698786.963519:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698786.963523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698786.963526:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698786.963531:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134656384:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698786.963578:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698786.963589:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698786/real 1787698786] req@ffff9d85b9d2ca80 x1874531134656384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698786.963599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134656384/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698786.963607:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698786.963609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698786.963613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134656384:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698786.963618:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134656512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698786.963634:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698786.963642:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698786/real 1787698786] req@ffff9d85b9d2f800 x1874531134656512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698786.963652:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134656512/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698786.963660:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698786.963663:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698786.963666:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134656512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698786.963671:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134656640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698786.963687:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698786.963696:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698786/real 1787698786] req@ffff9d85b9d2fb80 x1874531134656640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698786.963707:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134656640/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698786.963715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698786.963719:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698786.963722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134656640:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698787.539076:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698788.563085:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698789.587008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698790.611013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698791.635023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698792.083002:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698792.083013:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698792.083018:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698792.083028:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6693 00000100:00080000:0.0:1787698792.083039:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698792.083045:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698792.083082:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134656768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698792.083109:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698792.083113:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698792.083115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698792.083119:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6693 00000100:00080000:0.0:1787698792.083122:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698792.083126:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698792.083136:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698792.083146:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134656768:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698792.083152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134656896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698792.083163:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698792.083167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698792.083169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698792.083172:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6693 00000100:00080000:0.0:1787698792.083176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698792.083180:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698792.083185:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698792.083196:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134657024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698792.083210:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698792.083211:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698792.083217:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698792.083219:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698792.083224:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6693 00000100:00080000:0.0:1787698792.083228:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698792.083232:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698792.083237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698792.083248:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134657152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698792.083255:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698792/real 1787698792] req@ffff9d85b9d0b800 x1874531134656768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698792.083260:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698792.083267:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698792.083271:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698792.083277:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6693 00000100:00080000:0.0:1787698792.083281:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698792.083285:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134656768/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698792.083288:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698792.083302:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698792.083307:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698792.083311:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698792.083317:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134656768:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698792.083318:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134657280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698792.083330:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698792.083334:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698792.083336:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698792.083338:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656896:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698792.083348:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6693 00000100:00080000:0.0:1787698792.083356:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698792.083359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698792.083364:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698792.083370:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698792.083370:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698792/real 1787698792] req@ffff9d85b9d09f80 x1874531134656896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698792.083386:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134657408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698792.083395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134656896/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698792.083403:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698792.083404:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698792.083411:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698792.083411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698792.083416:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698792.083417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134656896:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698792.083426:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6693 00000100:00100000:1.0:1787698792.083427:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657024:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698792.083433:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698792.083439:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698792.083444:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698792.083457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698792.083470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698792/real 1787698792] req@ffff9d85b9d0a300 x1874531134657024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698792.083476:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134657536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698792.083493:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134657024/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698792.083503:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698792.083506:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698792.083510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698792.083517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698792.083537:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698792.083548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698792/real 1787698792] req@ffff9d85b9d09180 x1874531134657152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698792.083558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134657152/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698792.083563:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698792.083565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698792.083567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698792.083570:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134657280:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698792.083583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698792.083588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698792/real 1787698792] req@ffff9d85b9d09500 x1874531134657280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698792.083595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134657280/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698792.083599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698792.083600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698792.083602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134657280:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698792.083612:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134657408:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698792.083621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698792.083626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698792/real 1787698792] req@ffff9d85b9d08700 x1874531134657408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698792.083633:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134657408/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698792.083637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698792.083639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698792.083640:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134657408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698792.083643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134657536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698792.083660:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698792.083665:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698792/real 1787698792] req@ffff9d85b9d09c00 x1874531134657536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698792.083672:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134657536/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698792.083676:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698792.083677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698792.083679:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134657536:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698792.659068:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698793.683015:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698794.707081:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698795.731006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698796.754999:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698797.202962:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698797.202971:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698797.202975:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698797.202984:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6698 00000100:00080000:3.0:1787698797.202994:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698797.203000:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698797.203032:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134657664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698797.203059:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698797.203062:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698797.203063:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698797.203066:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6698 00000100:00080000:3.0:1787698797.203069:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698797.203071:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698797.203078:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698797.203088:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134657792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698797.203095:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698797.203098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698797.203100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698797.203102:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134657664:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698797.203105:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6698 00000100:00080000:3.0:1787698797.203111:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698797.203114:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698797.203117:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698797.203124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134657920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698797.203129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698797.203132:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698797.203133:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698797.203135:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6698 00000100:00080000:3.0:1787698797.203137:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698797.203139:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698797.203141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698797.203147:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134658048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698797.203152:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698797.203156:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698797.203156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698797.203158:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698797.203162:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6698 00000100:00080000:3.0:1787698797.203165:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698797.203168:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698797.203171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698797.203178:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134658176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698797.203184:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698797.203186:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698797.203187:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698797.203191:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698797/real 1787698797] req@ffff9d85c0fb1c00 x1874531134657664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698797.203191:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6698 00000100:00080000:3.0:1787698797.203196:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698797.203202:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698797.203206:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698797.203212:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134657664/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698797.203215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134658304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698797.203245:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698797.203246:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698797.203251:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698797.203253:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698797.203256:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134657664:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698797.203258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698797.203264:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6698 00000100:00080000:3.0:1787698797.203267:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698797.203270:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698797.203273:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698797.203276:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698797.203286:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134658432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698797.203293:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698797.203304:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698797/real 1787698797] req@ffff9d85c0fb0a80 x1874531134657792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698797.203313:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134657792/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698797.203318:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698797.203320:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698797.203323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698797.203326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698797.203335:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698797.203340:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698797/real 1787698797] req@ffff9d85c0fb3b80 x1874531134657920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698797.203347:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134657920/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698797.203351:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698797.203353:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698797.203355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134657920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698797.203358:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698797.203389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698797.203396:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698797/real 1787698797] req@ffff9d85c0fb0380 x1874531134658048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698797.203402:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134658048/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698797.203407:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698797.203409:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698797.203410:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698797.203413:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134658176:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698797.203423:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698797.203428:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698797/real 1787698797] req@ffff9d85c0fb2d80 x1874531134658176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698797.203434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134658176/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698797.203438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698797.203439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698797.203441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134658176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698797.203444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134658304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698797.203452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698797.203456:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698797/real 1787698797] req@ffff9d85c0fb3800 x1874531134658304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698797.203463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134658304/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698797.203468:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698797.203469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698797.203470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134658304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698797.203480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134658432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698797.203489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698797.203493:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698797/real 1787698797] req@ffff9d85c0fb0000 x1874531134658432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698797.203500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134658432/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698797.203504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698797.203506:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698797.203507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134658432:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698797.778969:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698798.803003:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698799.826979:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698800.850991:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698801.875031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698802.322986:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698802.322997:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698802.323001:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698802.323012:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6704 00000100:00080000:3.0:1787698802.323024:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698802.323031:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698802.323070:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134658560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698802.323097:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698802.323100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698802.323101:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698802.323105:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6704 00000100:00080000:3.0:1787698802.323107:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698802.323109:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698802.323117:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698802.323126:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134658688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698802.323133:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698802.323135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698802.323136:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698802.323138:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6704 00000100:00080000:3.0:1787698802.323141:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698802.323144:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698802.323148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698802.323156:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134658560:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698802.323156:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134658816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698802.323168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698802.323171:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698802.323172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698802.323174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6704 00000100:00080000:3.0:1787698802.323176:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698802.323178:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698802.323180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698802.323189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134658944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698802.323194:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698802.323196:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698802.323197:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698802.323199:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6704 00000100:00080000:3.0:1787698802.323201:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698802.323204:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698802.323209:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698802.323219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134659072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698802.323233:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698802.323237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787698802.323240:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698802.323241:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698802.323247:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6704 00000100:00080000:3.0:1787698802.323252:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698802.323257:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698802.323262:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698802.323272:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134659200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698802.323291:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698802.323293:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698802.323294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698802.323297:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6704 00000100:00080000:3.0:1787698802.323299:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698802.323302:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698802.323305:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698802.323313:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134659328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698802.323333:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698802/real 1787698802] req@ffff9d85c0fb2a00 x1874531134658560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698802.323352:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134658560/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698802.323364:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698802.323367:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698802.323372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134658560:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698802.323379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658688:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698802.323395:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698802.323403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698802/real 1787698802] req@ffff9d85c0fb2680 x1874531134658688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698802.323412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134658688/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698802.323420:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698802.323423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698802.323426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698802.323451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698802.323470:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698802.323478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698802/real 1787698802] req@ffff9d85c0fb0e00 x1874531134658816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698802.323486:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134658816/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698802.323491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698802.323493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698802.323494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698802.323498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698802.323508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698802.323513:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698802/real 1787698802] req@ffff9d85c0fb0000 x1874531134658944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698802.323520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134658944/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698802.323525:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698802.323526:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698802.323528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134658944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698802.323530:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659072:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698802.323540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698802.323545:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698802/real 1787698802] req@ffff9d85c0fb3800 x1874531134659072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698802.323551:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134659072/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698802.323555:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698802.323557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698802.323559:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698802.323561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698802.323569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698802.323574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698802/real 1787698802] req@ffff9d85c0fb2d80 x1874531134659200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698802.323580:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134659200/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698802.323585:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698802.323586:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698802.323587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698802.323590:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698802.323598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698802.323603:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698802/real 1787698802] req@ffff9d85c0fb0380 x1874531134659328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698802.323608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134659328/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698802.323612:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698802.323614:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698802.323615:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659328:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698802.899062:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698803.923044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698804.947047:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698805.971009:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698806.995042:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698807.442965:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698807.442977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698807.442981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698807.442990:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6709 00000100:00080000:1.0:1787698807.443002:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698807.443009:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698807.443051:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134659456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698807.443081:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698807.443084:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698807.443085:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698807.443088:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6709 00000100:00080000:1.0:1787698807.443090:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698807.443092:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698807.443101:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698807.443109:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134659584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698807.443115:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698807.443118:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698807.443119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698807.443121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6709 00000100:00080000:1.0:1787698807.443123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698807.443125:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698807.443129:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698807.443134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134659712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698807.443141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698807.443144:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698807.443145:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698807.443149:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6709 00000100:00100000:0.0:1787698807.443153:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134659456:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698807.443153:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698807.443156:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698807.443162:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698807.443169:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134659840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698807.443174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698807.443176:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698807.443178:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698807.443180:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6709 00000100:00080000:1.0:1787698807.443182:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698807.443184:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698807.443187:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698807.443211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134659968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698807.443218:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698807.443220:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698807.443222:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698807.443225:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6709 00000100:00080000:1.0:1787698807.443234:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698807.443237:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698807.443241:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698807.443247:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134660096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698807.443257:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698807.443259:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698807.443262:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698807.443265:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698807.443267:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6709 00000100:00080000:1.0:1787698807.443270:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698807.443272:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698807.443274:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698807.443281:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134660224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698807.443320:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698807/real 1787698807] req@ffff9d85b9d2fb80 x1874531134659456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698807.443341:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134659456/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698807.443352:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698807.443356:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698807.443360:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134659456:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698807.443369:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134659584:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698807.443392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698807.443403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698807/real 1787698807] req@ffff9d85b9d2f800 x1874531134659584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698807.443414:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134659584/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698807.443421:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698807.443424:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698807.443427:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134659584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698807.443451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134659712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698807.443471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698807.443482:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698807/real 1787698807] req@ffff9d85b9d2ca80 x1874531134659712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698807.443494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134659712/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698807.443502:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698807.443505:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698807.443509:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134659712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698807.443514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134659840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698807.443529:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698807.443536:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698807/real 1787698807] req@ffff9d85b9d2f480 x1874531134659840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698807.443543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134659840/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698807.443549:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698807.443550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698807.443552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134659840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698807.443555:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659968:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698807.443564:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698807.443570:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698807/real 1787698807] req@ffff9d85b9d2e300 x1874531134659968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698807.443576:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134659968/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698807.443581:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698807.443582:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698807.443584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134659968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698807.443586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698807.443594:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698807.443599:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698807/real 1787698807] req@ffff9d85b9d2c000 x1874531134660096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698807.443604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134660096/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698807.443613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698807.443615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698807.443617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698807.443620:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698807.443627:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698807.443632:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698807/real 1787698807] req@ffff9d85b9d2ce00 x1874531134660224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698807.443637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134660224/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698807.443642:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698807.443643:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698807.443645:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660224:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698808.019236:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698809.043030:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698810.067045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698811.091073:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698812.114988:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698812.562973:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698812.562983:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698812.562986:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698812.562995:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6714 00000100:00080000:1.0:1787698812.563004:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698812.563010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698812.563043:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134660352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698812.563071:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698812.563075:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698812.563076:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698812.563081:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6714 00000100:00080000:1.0:1787698812.563084:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698812.563087:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698812.563097:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698812.563124:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134660480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698812.563136:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698812.563140:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134660352:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698812.563141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698812.563145:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698812.563152:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6714 00000100:00080000:1.0:1787698812.563156:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698812.563160:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698812.563165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698812.563179:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134660608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698812.563188:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698812.563192:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698812.563194:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698812.563198:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6714 00000100:00080000:1.0:1787698812.563202:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698812.563207:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698812.563209:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698812.563216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698812.563227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134660736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698812.563236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698812.563240:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698812.563242:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698812.563247:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6714 00000100:00100000:0.0:1787698812.563251:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698812/real 1787698812] req@ffff9d85b9d2dc00 x1874531134660352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698812.563252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698812.563259:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698812.563267:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698812.563281:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134660352/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698812.563282:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134660864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698812.563309:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698812.563309:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698812.563316:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698812.563317:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698812.563320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134660352:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698812.563321:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698812.563330:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6714 00000100:00080000:1.0:1787698812.563333:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698812.563337:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698812.563342:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698812.563347:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134660480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698812.563356:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134660992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698812.563367:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698812.563368:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698812.563374:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698812.563376:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698812.563380:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698812/real 1787698812] req@ffff9d85b9d2c380 x1874531134660480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698812.563382:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6714 00000100:00080000:1.0:1787698812.563389:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698812.563394:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134660480/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698812.563396:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698812.563403:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698812.563406:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698812.563411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698812.563416:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134660480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698812.563426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134660608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698812.563426:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134661120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698812.563452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698812.563461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698812/real 1787698812] req@ffff9d85b9d2c700 x1874531134660608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698812.563471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134660608/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698812.563476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698812.563478:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698812.563480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134660608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698812.563484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134660736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698812.563494:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698812.563499:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698812/real 1787698812] req@ffff9d85b9d2ed80 x1874531134660736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698812.563506:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134660736/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698812.563511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698812.563512:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698812.563514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134660736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698812.563516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660864:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698812.563525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698812.563533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698812/real 1787698812] req@ffff9d85b9d2d500 x1874531134660864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698812.563551:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134660864/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698812.563559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698812.563563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698812.563566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698812.563582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698812.563596:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698812.563602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698812/real 1787698812] req@ffff9d85b9d2e680 x1874531134660992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698812.563609:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134660992/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698812.563622:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698812.563625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698812.563628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134660992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698812.563633:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134661120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698812.563651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698812.563661:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698812/real 1787698812] req@ffff9d85b9d2dc00 x1874531134661120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698812.563672:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134661120/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698812.563680:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698812.563683:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698812.563686:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134661120:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698813.139031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698814.162996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698815.186988:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698816.211079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698817.234964:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698817.682959:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698817.682969:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698817.682981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698817.682988:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6719 00000100:00080000:3.0:1787698817.682998:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698817.683004:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698817.683046:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134661248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698817.683066:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698817.683069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698817.683070:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698817.683073:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6719 00000100:00080000:3.0:1787698817.683076:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698817.683078:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698817.683085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698817.683097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134661376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698817.683108:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698817.683113:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698817.683115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698817.683119:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134661248:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698817.683120:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6719 00000100:00080000:3.0:1787698817.683128:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698817.683134:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698817.683138:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698817.683148:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134661504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698817.683155:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698817.683158:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698817.683159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698817.683163:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6719 00000100:00080000:3.0:1787698817.683167:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698817.683171:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698817.683173:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787698817.683178:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698817.683192:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134661632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698817.683203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698817.683208:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698817/real 1787698817] req@ffff9d85c0fb3b80 x1874531134661248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698817.683208:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698817.683211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698817.683215:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6719 00000100:00080000:3.0:1787698817.683220:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698817.683227:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698817.683235:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134661248/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698817.683236:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698817.683247:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698817.683252:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698817.683255:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134661248:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698817.683255:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134661760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698817.683268:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698817.683271:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698817.683273:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698817.683276:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6719 00000100:00080000:3.0:1787698817.683278:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698817.683282:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134661376:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787698817.683284:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698817.683290:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698817.683299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134661888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698817.683305:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698817.683308:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698817.683310:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698817.683312:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6719 00000800:00000100:0.0:1787698817.683316:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698817.683316:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698817.683320:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698817.683324:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698817.683331:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134662016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698817.683337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698817/real 1787698817] req@ffff9d85c0fb0a80 x1874531134661376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698817.683353:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134661376/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698817.683368:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698817.683371:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698817.683374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134661376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698817.683378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134661504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698817.683393:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698817.683400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698817/real 1787698817] req@ffff9d85c0fb1c00 x1874531134661504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698817.683407:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134661504/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698817.683411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698817.683413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698817.683415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134661504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698817.683418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134661632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698817.683427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698817.683431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698817/real 1787698817] req@ffff9d85c0fb1f80 x1874531134661632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698817.683438:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134661632/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698817.683442:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698817.683444:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698817.683446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134661632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698817.683449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134661760:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698817.683457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698817.683462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698817/real 1787698817] req@ffff9d85c0fb3100 x1874531134661760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698817.683474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134661760/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698817.683479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698817.683481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698817.683485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134661760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698817.683500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134661888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698817.683518:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698817.683528:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698817/real 1787698817] req@ffff9d85c0fb1880 x1874531134661888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698817.683539:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134661888/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698817.683548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698817.683550:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698817.683553:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134661888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698817.683558:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698817.683576:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698817.683586:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698817/real 1787698817] req@ffff9d85c0fb1500 x1874531134662016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698817.683597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134662016/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698817.683603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698817.683606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698817.683609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662016:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698818.259003:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698819.282968:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698820.307066:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698821.330972:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698822.355007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698822.803001:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698822.803009:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698822.803011:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698822.803020:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6724 00000100:00080000:3.0:1787698822.803028:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698822.803034:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698822.803062:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134662144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698822.803085:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698822.803088:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698822.803089:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698822.803092:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6724 00000100:00080000:3.0:1787698822.803094:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698822.803096:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698822.803102:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698822.803109:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134662272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698822.803116:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698822.803118:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698822.803119:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698822.803121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6724 00000100:00080000:3.0:1787698822.803123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698822.803125:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803130:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134662144:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787698822.803130:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698822.803139:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134662400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698822.803145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698822.803147:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698822.803149:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698822.803151:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6724 00000100:00080000:3.0:1787698822.803152:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698822.803154:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698822.803157:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698822.803165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134662528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698822.803178:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698822.803181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787698822.803182:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698822.803183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698822.803188:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6724 00000100:00080000:3.0:1787698822.803190:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698822.803193:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698822.803196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698822.803204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134662656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698822.803210:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698822.803212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698822.803214:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698822.803216:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6724 00000100:00080000:3.0:1787698822.803219:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698822.803220:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698822/real 1787698822] req@ffff9d85c0fb0700 x1874531134662144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698822.803223:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698822.803229:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698822.803240:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134662144/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698822.803241:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134662784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698822.803268:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698822.803270:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698822.803271:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698822.803275:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698822.803277:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698822.803280:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6724 00000100:00100000:0.0:1787698822.803284:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134662144:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698822.803284:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698822.803289:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803295:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134662272:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787698822.803296:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698822.803310:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134662912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698822.803316:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803326:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698822/real 1787698822] req@ffff9d85c0fb3480 x1874531134662272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698822.803335:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134662272/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698822.803340:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698822.803342:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698822.803345:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134662272:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698822.803362:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134662400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698822.803372:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803377:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698822/real 1787698822] req@ffff9d85c0fb2300 x1874531134662400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698822.803384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134662400/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698822.803388:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698822.803390:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698822.803391:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134662400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698822.803395:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134662528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698822.803403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803408:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698822/real 1787698822] req@ffff9d85c0fb1500 x1874531134662528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698822.803414:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134662528/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698822.803418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698822.803420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698822.803421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134662528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698822.803424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662656:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698822.803440:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803446:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698822/real 1787698822] req@ffff9d85c0fb1880 x1874531134662656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698822.803453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134662656/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698822.803458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698822.803459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698822.803461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698822.803464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698822.803473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698822/real 1787698822] req@ffff9d85c0fb3100 x1874531134662784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698822.803483:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134662784/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698822.803491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698822.803493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698822.803495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698822.803497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698822.803505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698822.803510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698822/real 1787698822] req@ffff9d85c0fb1f80 x1874531134662912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698822.803518:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134662912/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698822.803525:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698822.803528:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698822.803531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134662912:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698823.379051:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698824.403011:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698825.427030:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698826.451051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698827.475029:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698827.922988:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698827.922999:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698827.923003:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698827.923013:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6729 00000100:00080000:0.0:1787698827.923024:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698827.923031:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698827.923066:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134663040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698827.923094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698827.923098:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698827.923100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698827.923104:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6729 00000100:00080000:0.0:1787698827.923107:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698827.923111:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698827.923119:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698827.923129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134663168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698827.923135:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698827.923138:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698827.923139:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698827.923141:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6729 00000100:00080000:0.0:1787698827.923143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698827.923145:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698827.923148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698827.923152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134663296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698827.923158:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698827.923161:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698827.923162:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698827.923164:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6729 00000100:00080000:0.0:1787698827.923166:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698827.923168:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698827.923171:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698827.923177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134663424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698827.923182:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134663040:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698827.923185:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698827.923191:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698827.923192:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698827.923195:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6729 00000100:00080000:0.0:1787698827.923197:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698827.923199:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698827.923201:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698827.923209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134663552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698827.923213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698827.923216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698827.923217:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698827.923219:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6729 00000100:00080000:0.0:1787698827.923221:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698827.923224:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698827.923226:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698827.923231:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134663680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698827.923236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698827.923237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698827.923239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698827.923241:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6729 00000100:00080000:0.0:1787698827.923243:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698827.923245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698827.923248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698827.923252:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698827.923256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134663808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698827.923307:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698827/real 1787698827] req@ffff9d85b9d0b480 x1874531134663040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698827.923323:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134663040/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698827.923332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698827.923336:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698827.923339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134663040:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698827.923346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134663168:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698827.923371:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698827.923382:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698827/real 1787698827] req@ffff9d85b9d0a680 x1874531134663168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698827.923393:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134663168/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698827.923401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698827.923403:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698827.923406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134663168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698827.923412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134663296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698827.923432:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698827.923442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698827/real 1787698827] req@ffff9d85b9d0bb80 x1874531134663296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698827.923452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134663296/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698827.923458:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698827.923462:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698827.923465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134663296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698827.923487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134663424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698827.923514:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698827.923525:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698827/real 1787698827] req@ffff9d85b9d0ad80 x1874531134663424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698827.923535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134663424/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698827.923542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698827.923545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698827.923548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134663424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698827.923553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134663552:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698827.923569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698827.923578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698827/real 1787698827] req@ffff9d85b9d08000 x1874531134663552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698827.923592:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134663552/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698827.923599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698827.923601:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698827.923604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134663552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698827.923608:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134663680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698827.923622:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698827.923630:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698827/real 1787698827] req@ffff9d85b9d08a80 x1874531134663680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698827.923639:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134663680/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698827.923645:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698827.923647:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698827.923650:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134663680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698827.923654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134663808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698827.923669:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698827.923678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698827/real 1787698827] req@ffff9d85b9d08e00 x1874531134663808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698827.923688:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134663808/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698827.923701:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698827.923704:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698827.923707:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134663808:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698828.498987:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698829.523049:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698830.547067:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698831.571050:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698832.595023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698833.042951:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698833.042958:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698833.042961:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698833.042977:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6734 00000100:00080000:2.0:1787698833.042991:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698833.042999:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698833.043032:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134663936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698833.043054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698833.043058:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698833.043060:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698833.043064:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6734 00000100:00080000:2.0:1787698833.043067:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698833.043070:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698833.043078:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698833.043086:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134663936:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698833.043091:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134664064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698833.043103:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698833.043107:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698833.043109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698833.043112:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6734 00000100:00080000:2.0:1787698833.043116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698833.043120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698833.043125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698833.043136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134664192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698833.043146:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698833.043150:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698833.043151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698833.043153:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698833.043158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6734 00000100:00080000:2.0:1787698833.043161:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698833.043163:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698833.043166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698833.043174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134664320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698833.043179:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698833.043182:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698833.043183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698833.043184:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6734 00000100:00080000:2.0:1787698833.043186:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698833.043188:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698833.043191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698833.043201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134664448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698833.043202:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698833/real 1787698833] req@ffff9d85babd4380 x1874531134663936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698833.043216:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698833.043223:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698833.043224:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698833.043227:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6734 00000100:00080000:2.0:1787698833.043230:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698833.043234:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134663936/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698833.043235:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698833.043243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698833.043251:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698833.043257:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787698833.043258:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134664576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698833.043261:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134663936:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698833.043274:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698833.043277:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698833.043278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698833.043282:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6734 00000100:00100000:1.0:1787698833.043285:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664064:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698833.043286:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698833.043293:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698833.043297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698833.043308:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134664704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698833.043311:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698833.043323:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698833/real 1787698833] req@ffff9d85babd6680 x1874531134664064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698833.043337:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134664064/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698833.043345:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698833.043348:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698833.043351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698833.043357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698833.043375:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698833.043384:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698833/real 1787698833] req@ffff9d85babd6a00 x1874531134664192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698833.043394:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134664192/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698833.043400:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698833.043403:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698833.043406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698833.043410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698833.043427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698833.043437:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698833/real 1787698833] req@ffff9d85babd5180 x1874531134664320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698833.043448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134664320/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698833.043455:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698833.043458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698833.043461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698833.043465:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134664448:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698833.043480:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698833.043488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698833/real 1787698833] req@ffff9d85babd7100 x1874531134664448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698833.043499:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134664448/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698833.043507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698833.043509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698833.043512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134664448:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698833.043517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134664576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698833.043529:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698833.043537:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698833/real 1787698833] req@ffff9d85babd4a80 x1874531134664576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698833.043547:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134664576/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698833.043554:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698833.043557:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698833.043560:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134664576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698833.043574:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134664704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698833.043587:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698833.043594:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698833/real 1787698833] req@ffff9d85babd7b80 x1874531134664704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698833.043604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134664704/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698833.043613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698833.043615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698833.043617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134664704:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698833.619009:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698834.643015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698835.667060:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698836.691039:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698837.715001:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698838.162944:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698838.162954:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698838.162957:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698838.162965:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6739 00000100:00080000:2.0:1787698838.162974:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698838.162978:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698838.163005:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134664832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698838.163025:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698838.163028:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698838.163029:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698838.163033:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6739 00000100:00080000:2.0:1787698838.163036:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698838.163040:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698838.163048:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698838.163058:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134664960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698838.163065:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134664832:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698838.163066:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698838.163069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698838.163071:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698838.163074:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6739 00000100:00080000:2.0:1787698838.163076:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698838.163079:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698838.163084:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698838.163091:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134665088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698838.163097:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698838.163100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698838.163101:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698838.163104:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6739 00000100:00080000:2.0:1787698838.163106:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698838.163109:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698838.163113:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698838.163114:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698838.163121:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134665216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698838.163128:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698838.163130:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698838.163132:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698838.163134:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6739 00000100:00080000:2.0:1787698838.163137:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698838.163140:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698838.163144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698838.163152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134665344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698838.163157:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698838/real 1787698838] req@ffff9d85babd4e00 x1874531134664832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698838.163160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698838.163163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698838.163165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698838.163169:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6739 00000100:00080000:2.0:1787698838.163172:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698838.163175:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134664832/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698838.163175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698838.163179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698838.163185:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698838.163186:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134665472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698838.163188:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698838.163192:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134664832:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698838.163199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698838.163201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698838.163203:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698838.163205:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6739 00000100:00080000:2.0:1787698838.163208:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698838.163210:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664960:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698838.163211:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698838.163214:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698838.163222:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134665600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698838.163228:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698838.163238:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698838/real 1787698838] req@ffff9d85babd4700 x1874531134664960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698838.163249:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134664960/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698838.163257:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698838.163260:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698838.163263:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134664960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698838.163269:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698838.163282:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698838.163289:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698838/real 1787698838] req@ffff9d85babd5f80 x1874531134665088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698838.163296:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134665088/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698838.163300:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698838.163302:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698838.163304:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698838.163307:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698838.163315:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698838.163320:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698838/real 1787698838] req@ffff9d85babd7b80 x1874531134665216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698838.163326:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134665216/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698838.163330:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698838.163332:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698838.163333:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698838.163336:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134665344:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698838.163356:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698838.163362:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698838/real 1787698838] req@ffff9d85babd4a80 x1874531134665344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698838.163369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134665344/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698838.163373:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698838.163375:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698838.163376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134665344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698838.163379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134665472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698838.163388:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698838.163393:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698838/real 1787698838] req@ffff9d85babd7100 x1874531134665472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698838.163399:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134665472/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698838.163403:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698838.163405:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698838.163406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134665472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698838.163416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134665600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698838.163425:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698838.163429:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698838/real 1787698838] req@ffff9d85babd5180 x1874531134665600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698838.163435:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134665600/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698838.163443:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698838.163445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698838.163446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134665600:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698838.738981:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698839.762987:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698840.787040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698841.811016:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698842.835012:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698843.282971:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698843.282979:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698843.282982:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698843.282990:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6745 00000100:00080000:0.0:1787698843.282998:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698843.283003:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698843.283039:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134665728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698843.283064:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698843.283068:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698843.283070:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698843.283073:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6745 00000100:00080000:0.0:1787698843.283077:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698843.283081:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698843.283090:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698843.283103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134665728:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698843.283107:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134665856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698843.283124:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698843.283128:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698843.283130:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698843.283134:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6745 00000100:00080000:0.0:1787698843.283147:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698843.283152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698843.283158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698843.283169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134665984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698843.283178:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698843.283182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698843.283184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698843.283189:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665856:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698843.283190:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6745 00000100:00080000:0.0:1787698843.283196:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698843.283201:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698843.283205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698843.283217:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134666112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698843.283218:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665984:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698843.283232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698843.283235:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698843.283237:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698843.283241:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134666112:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698843.283243:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6745 00000100:00080000:0.0:1787698843.283248:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698843.283252:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698843.283257:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698843.283267:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134666240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698843.283275:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698843.283277:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698843.283278:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698843.283284:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6745 00000100:00100000:1.0:1787698843.283284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134666240:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698843.283289:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698843.283293:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698843.283296:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698843.283304:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134666368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698843.283316:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698843.283319:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698843.283322:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698843.283326:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6745 00000100:00080000:0.0:1787698843.283330:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698843.283332:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134666368:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787698843.283337:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698843.283344:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698843.283358:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134666496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698843.283407:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134666496:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698843.859006:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698844.883003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698845.906995:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698846.355109:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698846.355142:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698846.355155:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698846.355188:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698846.355197:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698846.355207:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698846.355215:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698846.355225:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698846.355237:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698846.355241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698843/real 1787698846] req@ffff9d85b9d0bb80 x1874531134666240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698846.355278:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134666240/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698846.355292:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698846.355298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698846.355301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134666240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698846.355308:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698843/real 1787698846] req@ffff9d85b9d0ad80 x1874531134666112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698846.355318:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134666112/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698846.355334:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698846.355339:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698846.355343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134666112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698846.355351:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698843/real 1787698846] req@ffff9d85b9d08a80 x1874531134665856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698846.355363:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134665856/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698846.355371:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698846.355375:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698846.355379:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698846.355387:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698843/real 1787698846] req@ffff9d85b9d08000 x1874531134665984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698846.355399:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134665984/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698846.355407:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698846.355411:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698846.355415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134665984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698846.355425:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698843/real 1787698846] req@ffff9d85b9d08e00 x1874531134665728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698846.355441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134665728/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698846.355450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698846.355454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698846.355458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134665728:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698846.355496:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698843/real 1787698846] req@ffff9d85b9d0b480 x1874531134666496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698846.355509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134666496/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698846.355520:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698846.355524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698846.355538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134666496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698846.355545:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698843/real 1787698846] req@ffff9d85b9d0a680 x1874531134666368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698846.355556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134666368/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698846.355565:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698846.355570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698846.355573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134666368:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698846.930999:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698847.954995:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698848.403002:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698848.403011:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698848.403016:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698848.403026:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6750 00000100:00080000:2.0:1787698848.403036:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698848.403044:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698848.403077:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134666624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698848.403105:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698848.403109:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698848.403111:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698848.403116:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6750 00000100:00080000:2.0:1787698848.403119:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698848.403123:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698848.403132:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698848.403148:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134666752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698848.403155:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134666624:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698848.403162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698848.403166:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698848.403170:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698848.403176:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6750 00000100:00080000:2.0:1787698848.403181:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698848.403197:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698848.403203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698848.403216:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134666880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698848.403233:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698848.403236:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698848.403237:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698848.403241:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6750 00000800:00000100:1.0:1787698848.403247:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698848.403248:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698848.403254:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698848.403260:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698848.403271:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134667008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698848.403279:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698848.403282:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698848.403284:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698848.403288:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6750 00000100:00080000:2.0:1787698848.403292:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698848.403297:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698848.403302:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698848.403312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134667136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698848.403321:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698848.403325:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698848/real 1787698848] req@ffff9d85babd6a00 x1874531134666624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698848.403326:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698848.403330:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698848.403340:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6750 00000100:00080000:2.0:1787698848.403345:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698848.403349:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698848.403354:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698848.403355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134666624/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698848.403369:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134667264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698848.403370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698848.403380:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787698848.403388:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698848.403389:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134666624:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698848.403397:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698848.403400:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698848.403406:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6750 00000100:00080000:2.0:1787698848.403410:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698848.403414:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698848.403419:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698848.403427:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134666752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698848.403434:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134667392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698848.403457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698848.403475:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698848/real 1787698848] req@ffff9d85babd6680 x1874531134666752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698848.403488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134666752/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698848.403512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698848.403516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698848.403520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134666752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698848.403527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134666880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698848.403548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698848.403561:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698848/real 1787698848] req@ffff9d85babd4380 x1874531134666880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698848.403580:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134666880/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698848.403590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698848.403593:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698848.403596:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134666880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698848.403602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698848.403622:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698848.403633:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698848/real 1787698848] req@ffff9d85babd6300 x1874531134667008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698848.403647:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134667008/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698848.403657:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698848.403661:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698848.403665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698848.403671:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134667136:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698848.403691:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698848.403703:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698848/real 1787698848] req@ffff9d85babd7480 x1874531134667136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698848.403714:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134667136/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698848.403729:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698848.403732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698848.403735:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134667136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698848.403741:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134667264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698848.403761:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698848.403772:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698848/real 1787698848] req@ffff9d85babd5c00 x1874531134667264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698848.403783:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134667264/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698848.403791:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698848.403795:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698848.403798:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134667264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698848.403817:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134667392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698848.403836:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698848.403848:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698848/real 1787698848] req@ffff9d85babd4000 x1874531134667392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698848.403894:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134667392/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698848.403903:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698848.403905:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698848.403909:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134667392:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698848.979001:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698850.003015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698851.027032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698852.051020:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698853.075023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698853.522941:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698853.522949:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698853.522952:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698853.522960:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6755 00000100:00080000:2.0:1787698853.522970:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698853.522975:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698853.523004:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134667520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698853.523025:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698853.523027:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698853.523029:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698853.523031:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6755 00000100:00080000:2.0:1787698853.523033:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698853.523035:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698853.523043:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698853.523051:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134667648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698853.523057:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698853.523060:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698853.523061:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698853.523063:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6755 00000100:00080000:2.0:1787698853.523065:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698853.523067:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698853.523071:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698853.523077:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134667520:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698853.523078:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134667776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698853.523090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698853.523092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698853.523093:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698853.523095:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6755 00000100:00080000:2.0:1787698853.523098:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698853.523099:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698853.523102:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698853.523111:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134667904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698853.523117:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698853.523119:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698853.523120:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698853.523121:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6755 00000100:00080000:2.0:1787698853.523123:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698853.523126:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698853.523128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698853.523134:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134668032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698853.523140:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698853.523144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698853.523147:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698853.523149:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698853.523153:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6755 00000100:00080000:2.0:1787698853.523156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698853.523158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698853.523161:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698853.523166:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134668160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698853.523175:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698853.523178:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698853.523179:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698853.523184:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6755 00000100:00100000:1.0:1787698853.523185:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698853/real 1787698853] req@ffff9d85babd7800 x1874531134667520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698853.523186:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698853.523190:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698853.523195:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698853.523205:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134668288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698853.523214:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134667520/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698853.523225:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698853.523229:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698853.523233:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134667520:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698853.523241:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667648:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698853.523263:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698853.523273:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698853/real 1787698853] req@ffff9d85babd6d80 x1874531134667648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698853.523284:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134667648/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698853.523292:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698853.523295:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698853.523298:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698853.523320:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698853.523336:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698853.523348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698853/real 1787698853] req@ffff9d85babd5500 x1874531134667776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698853.523359:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134667776/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698853.523367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698853.523370:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698853.523373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698853.523378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698853.523392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698853.523402:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698853/real 1787698853] req@ffff9d85babd4000 x1874531134667904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698853.523428:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134667904/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698853.523435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698853.523438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698853.523441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134667904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698853.523447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668032:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698853.523464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698853.523473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698853/real 1787698853] req@ffff9d85babd5c00 x1874531134668032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698853.523484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134668032/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698853.523492:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698853.523495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698853.523498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698853.523503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698853.523518:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698853.523527:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698853/real 1787698853] req@ffff9d85babd7480 x1874531134668160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698853.523537:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134668160/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698853.523544:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698853.523547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698853.523550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698853.523556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698853.523570:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698853.523579:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698853/real 1787698853] req@ffff9d85babd6300 x1874531134668288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698853.523589:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134668288/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698853.523597:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698853.523599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698853.523603:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668288:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698854.099032:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698855.122954:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698856.147008:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698857.170980:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698858.195031:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698858.642937:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698858.642947:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698858.642950:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698858.642960:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6760 00000100:00080000:1.0:1787698858.642970:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698858.642977:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698858.643009:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134668416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698858.643036:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698858.643039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698858.643041:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698858.643045:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6760 00000100:00080000:1.0:1787698858.643049:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698858.643053:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698858.643062:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698858.643070:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134668416:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698858.643077:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134668544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698858.643089:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698858.643093:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698858.643095:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698858.643098:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6760 00000100:00080000:1.0:1787698858.643113:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698858.643120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698858.643121:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787698858.643129:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698858.643142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134668672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698858.643151:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698858.643154:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698858/real 1787698858] req@ffff9d85b9d2ce00 x1874531134668416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698858.643156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698858.643161:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698858.643170:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6760 00000100:00100000:0.0:1787698858.643173:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134668416/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698858.643176:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698858.643185:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698858.643185:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698858.643193:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:1.0:1787698858.643194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698858.643196:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134668416:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698858.643210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134668800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698858.643218:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134668544:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698858.643223:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698858.643228:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698858.643230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698858.643235:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6760 00000800:00000100:0.0:1787698858.643238:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698858.643241:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698858.643248:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698858.643250:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698858/real 1787698858] req@ffff9d85b9d2c000 x1874531134668544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698858.643256:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698858.643260:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134668544/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698858.643267:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698858.643270:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134668928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698858.643271:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698858.643277:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134668544:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698858.643286:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134668672:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787698858.643292:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698858.643297:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698858.643299:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698858.643303:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698858.643307:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6760 00000100:00080000:1.0:1787698858.643313:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698858.643314:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698858/real 1787698858] req@ffff9d85b9d2e300 x1874531134668672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698858.643340:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134668672/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698858.643345:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698858.643347:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698858.643349:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134668672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698858.643358:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134668800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698858.643372:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698858.643378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698858/real 1787698858] req@ffff9d85b9d2f480 x1874531134668800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698858.643397:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134668800/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698858.643405:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698858.643408:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698858.643411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134668800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698858.643417:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668928:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698858.643434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698858.643442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698858/real 1787698858] req@ffff9d85b9d2ca80 x1874531134668928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698858.643446:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698858.643452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134668928/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698858.643455:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698858.643461:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698858.643464:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698858.643466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134668928:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698858.643473:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134669056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698858.643487:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698858.643491:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698858.643493:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698858.643497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669056:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787698858.643501:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6760 00000100:00080000:1.0:1787698858.643505:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698858.643510:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698858.643515:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698858.643520:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698858.643529:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134669184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698858.643531:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698858/real 1787698858] req@ffff9d85b9d2f800 x1874531134669056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698858.643556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134669056/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698858.643562:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698858.643565:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698858.643566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698858.643575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698858.643585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698858.643590:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698858/real 1787698858] req@ffff9d85b9d2fb80 x1874531134669184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698858.643597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134669184/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698858.643601:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698858.643602:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698858.643604:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669184:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698859.219031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698860.243001:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698861.267074:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698862.291019:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698863.315006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698863.763000:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698863.763010:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698863.763014:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698863.763023:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6765 00000100:00080000:0.0:1787698863.763034:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698863.763041:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698863.763076:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134669312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698863.763104:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698863.763107:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698863.763109:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698863.763113:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6765 00000100:00080000:0.0:1787698863.763116:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698863.763120:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698863.763128:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698863.763140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134669440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698863.763148:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134669312:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698863.763152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698863.763156:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698863.763158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698863.763162:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6765 00000100:00080000:0.0:1787698863.763165:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698863.763169:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698863.763174:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698863.763184:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134669568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698863.763193:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698863.763196:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698863.763198:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698863.763201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6765 00000100:00080000:0.0:1787698863.763204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698863.763209:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698863.763211:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698863.763214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698863.763226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134669696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698863.763249:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698863.763252:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698863.763254:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698863.763257:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6765 00000100:00080000:0.0:1787698863.763260:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698863.763264:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698863.763268:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698863.763278:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698863/real 1787698863] req@ffff9d85b9d09c00 x1874531134669312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698863.763281:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134669824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698863.763303:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698863.763306:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698863.763309:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698863.763311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134669312/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698863.763315:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6765 00000100:00080000:0.0:1787698863.763320:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698863.763327:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698863.763333:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698863.763344:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134669952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698863.763349:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698863.763355:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698863.763359:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698863.763361:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698863.763361:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698863.763369:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6765 00000100:00100000:1.0:1787698863.763372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134669312:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698863.763374:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698863.763380:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698863.763385:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698863.763395:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134670080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698863.763402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134669440:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698863.763431:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698863.763444:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698863/real 1787698863] req@ffff9d85b9d08700 x1874531134669440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698863.763458:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134669440/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698863.763466:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698863.763470:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698863.763473:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134669440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698863.763481:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134669568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698863.763500:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698863.763510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698863/real 1787698863] req@ffff9d85b9d09500 x1874531134669568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698863.763521:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134669568/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698863.763529:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698863.763532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698863.763535:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134669568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698863.763540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134669696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698863.763557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698863.763566:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698863/real 1787698863] req@ffff9d85b9d09180 x1874531134669696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698863.763576:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134669696/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698863.763584:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698863.763587:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698863.763590:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134669696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698863.763595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669824:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698863.763614:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698863.763625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698863/real 1787698863] req@ffff9d85b9d0a300 x1874531134669824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698863.763636:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134669824/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698863.763644:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698863.763647:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698863.763650:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669824:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698863.763655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698863.763673:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698863.763683:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698863/real 1787698863] req@ffff9d85b9d09f80 x1874531134669952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698863.763694:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134669952/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698863.763702:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698863.763705:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698863.763708:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134669952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698863.763726:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698863.763743:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698863.763754:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698863/real 1787698863] req@ffff9d85b9d0b800 x1874531134670080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698863.763765:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134670080/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698863.763772:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698863.763776:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698863.763779:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670080:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698864.339012:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698865.363023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698866.386997:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698867.410991:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698868.435000:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698868.882973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698868.882983:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698868.882987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698868.883003:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6770 00000100:00080000:2.0:1787698868.883014:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698868.883020:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698868.883058:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134670208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698868.883083:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698868.883086:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698868.883089:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698868.883093:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6770 00000100:00080000:2.0:1787698868.883097:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698868.883100:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698868.883109:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698868.883121:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134670336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698868.883129:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134670208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698868.883135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698868.883139:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698868.883142:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698868.883171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6770 00000100:00080000:2.0:1787698868.883176:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698868.883181:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698868.883187:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698868.883199:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134670464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698868.883206:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698868.883220:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698868.883224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698868.883226:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698868.883231:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6770 00000100:00080000:2.0:1787698868.883235:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698868.883239:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698868.883244:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698868.883253:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698868/real 1787698868] req@ffff9d85babd4380 x1874531134670208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698868.883258:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134670592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698868.883275:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698868.883283:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698868.883285:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698868.883289:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6770 00000100:00100000:1.0:1787698868.883293:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134670208/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698868.883294:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698868.883301:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698868.883308:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698868.883322:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698868.883328:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787698868.883329:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134670720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698868.883334:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134670208:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698868.883349:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698868.883352:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698868.883355:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698868.883360:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134670336:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698868.883361:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6770 00000100:00080000:2.0:1787698868.883367:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698868.883373:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698868.883379:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698868.883389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698868.883393:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134670848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698868.883405:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698868/real 1787698868] req@ffff9d85babd6680 x1874531134670336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698868.883406:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698868.883414:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698868.883419:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698868.883426:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6770 00000100:00100000:1.0:1787698868.883430:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134670336/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698868.883432:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698868.883439:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698868.883444:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698868.883454:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698868.883459:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787698868.883459:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134670976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698868.883466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134670336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698868.883479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134670464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698868.883501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698868.883511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698868/real 1787698868] req@ffff9d85babd6a00 x1874531134670464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698868.883522:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134670464/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698868.883530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698868.883533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698868.883536:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134670464:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698868.883542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134670592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698868.883557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698868.883565:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698868/real 1787698868] req@ffff9d85babd5180 x1874531134670592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698868.883575:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134670592/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698868.883582:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698868.883585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698868.883588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134670592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698868.883593:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670720:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698868.883607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698868.883616:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698868/real 1787698868] req@ffff9d85babd7100 x1874531134670720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698868.883626:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134670720/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698868.883633:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698868.883636:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698868.883639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698868.883654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698868.883672:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698868.883682:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698868/real 1787698868] req@ffff9d85babd4a80 x1874531134670848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698868.883694:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134670848/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698868.883702:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698868.883706:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698868.883708:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698868.883714:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698868.883730:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698868.883740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698868/real 1787698868] req@ffff9d85babd7b80 x1874531134670976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698868.883751:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134670976/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698868.883759:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698868.883762:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698868.883765:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134670976:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698869.458975:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698870.483039:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698871.506994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698872.531045:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698873.555005:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698874.002967:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698874.002976:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698874.002980:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698874.002990:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6775 00000100:00080000:0.0:1787698874.003000:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698874.003006:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698874.003040:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134671104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698874.003076:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698874.003079:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698874.003081:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698874.003085:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6775 00000100:00080000:0.0:1787698874.003088:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698874.003092:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698874.003101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698874.003113:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134671232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698874.003120:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134671104:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698874.003126:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698874.003130:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698874.003131:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698874.003135:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6775 00000100:00080000:0.0:1787698874.003139:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698874.003142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698874.003147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698874.003158:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134671360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698874.003166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698874.003169:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698874.003171:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698874.003174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6775 00000100:00080000:0.0:1787698874.003178:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698874.003183:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698874.003184:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787698874.003189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698874.003200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134671488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698874.003208:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698874.003211:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698874.003213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698874.003217:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6775 00000100:00080000:0.0:1787698874.003221:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698874.003224:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698874.003230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698874.003230:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698874/real 1787698874] req@ffff9d85b9d0b800 x1874531134671104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698874.003244:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134671616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698874.003253:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134671104/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698874.003254:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698874.003269:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698874.003273:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698874.003282:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6775 00000100:00080000:1.0:1787698874.003283:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698874.003285:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698874.003291:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698874.003293:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698874.003300:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698874.003301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134671104:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698874.003315:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134671744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698874.003329:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698874.003331:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698874.003333:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698874.003337:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6775 00000100:00080000:0.0:1787698874.003340:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698874.003342:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134671232:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787698874.003346:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698874.003359:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698874.003373:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134671872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698874.003373:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698874.003392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698874/real 1787698874] req@ffff9d85b9d09f80 x1874531134671232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698874.003408:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134671232/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698874.003417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698874.003420:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698874.003423:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134671232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698874.003429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134671360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698874.003450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698874.003460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698874/real 1787698874] req@ffff9d85b9d0a300 x1874531134671360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698874.003471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134671360/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698874.003479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698874.003482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698874.003485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134671360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698874.003491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134671488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698874.003507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698874.003516:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698874/real 1787698874] req@ffff9d85b9d09180 x1874531134671488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698874.003526:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134671488/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698874.003534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698874.003537:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698874.003540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134671488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698874.003545:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134671616:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698874.003563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698874.003574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698874/real 1787698874] req@ffff9d85b9d09500 x1874531134671616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698874.003586:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134671616/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698874.003594:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698874.003597:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698874.003600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134671616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698874.003606:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134671744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698874.003624:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698874.003634:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698874/real 1787698874] req@ffff9d85b9d08700 x1874531134671744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698874.003645:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134671744/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698874.003652:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698874.003655:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698874.003658:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134671744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698874.003675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134671872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698874.003693:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698874.003703:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698874/real 1787698874] req@ffff9d85b9d09c00 x1874531134671872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698874.003715:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134671872/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698874.003722:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698874.003726:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698874.003729:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134671872:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698874.579027:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698875.603016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698876.627001:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698877.651077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698878.675023:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698879.122955:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698879.122962:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698879.122965:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698879.122983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6780 00000100:00080000:0.0:1787698879.122992:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698879.122997:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698879.123028:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134672000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698879.123057:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698879.123060:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698879.123062:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698879.123066:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6780 00000100:00080000:0.0:1787698879.123070:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698879.123073:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698879.123082:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698879.123092:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134672128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698879.123099:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698879.123101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698879.123103:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698879.123105:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6780 00000100:00080000:0.0:1787698879.123107:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698879.123110:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698879.123113:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698879.123121:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134672256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698879.123121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134672000:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698879.123135:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698879.123137:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698879.123138:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698879.123140:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6780 00000100:00080000:0.0:1787698879.123142:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698879.123144:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698879.123147:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698879.123154:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134672384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698879.123160:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698879.123161:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698879.123163:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698879.123165:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6780 00000100:00080000:0.0:1787698879.123186:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698879.123189:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698879.123191:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698879.123198:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698879.123200:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134672512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698879.123207:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698879.123210:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698879.123211:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698879.123214:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6780 00000100:00080000:0.0:1787698879.123219:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698879.123221:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698879.123223:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698879.123230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134672640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698879.123236:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698879.123238:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698879.123239:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698879.123243:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6780 00000100:00100000:1.0:1787698879.123244:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698879/real 1787698879] req@ffff9d85b9d0a680 x1874531134672000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698879.123245:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698879.123250:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698879.123255:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698879.123266:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134672768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698879.123274:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134672000/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698879.123291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698879.123296:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698879.123301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134672000:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698879.123308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134672128:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698879.123328:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698879.123339:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698879/real 1787698879] req@ffff9d85b9d0b480 x1874531134672128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698879.123350:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134672128/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698879.123358:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698879.123361:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698879.123364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134672128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698879.123384:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134672256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698879.123401:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698879.123411:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698879/real 1787698879] req@ffff9d85b9d08e00 x1874531134672256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698879.123422:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134672256/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698879.123429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698879.123431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698879.123434:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134672256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698879.123441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134672384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698879.123467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698879.123477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698879/real 1787698879] req@ffff9d85b9d08000 x1874531134672384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698879.123487:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134672384/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698879.123495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698879.123498:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698879.123500:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134672384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698879.123505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134672512:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698879.123524:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698879.123534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698879/real 1787698879] req@ffff9d85b9d08a80 x1874531134672512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698879.123545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134672512/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698879.123553:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698879.123555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698879.123558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134672512:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698879.123563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134672640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698879.123580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698879.123590:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698879/real 1787698879] req@ffff9d85b9d0ad80 x1874531134672640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698879.123602:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134672640/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698879.123610:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698879.123613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698879.123616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134672640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698879.123620:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134672768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698879.123636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698879.123660:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698879/real 1787698879] req@ffff9d85b9d0bb80 x1874531134672768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698879.123672:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134672768/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698879.123679:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698879.123682:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698879.123685:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134672768:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698879.699034:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698880.723024:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698881.747051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698882.771001:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698883.795080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698884.242976:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698884.242987:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698884.242990:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698884.242997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6786 00000100:00080000:0.0:1787698884.243006:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698884.243012:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698884.243043:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134672896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698884.243063:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698884.243066:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698884.243067:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698884.243070:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6786 00000100:00080000:0.0:1787698884.243072:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698884.243073:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698884.243079:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698884.243087:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134673024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698884.243093:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698884.243095:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698884.243097:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698884.243099:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6786 00000100:00080000:0.0:1787698884.243101:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698884.243103:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698884.243107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698884.243109:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134672896:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698884.243118:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134673152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698884.243128:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698884.243131:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698884.243133:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698884.243136:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6786 00000100:00080000:0.0:1787698884.243140:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698884.243144:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698884.243153:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698884.243166:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134673280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698884.243170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698884.243180:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698884.243183:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698884.243185:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698884.243190:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6786 00000100:00080000:0.0:1787698884.243193:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698884.243198:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698884.243203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698884.243209:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698884/real 1787698884] req@ffff9d85b9d0bb80 x1874531134672896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698884.243217:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134673408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698884.243226:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134672896/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698884.243232:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698884.243242:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698884.243246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698884.243249:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698884.243252:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6786 00000100:00080000:1.0:1787698884.243255:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787698884.243260:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698884.243260:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134672896:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787698884.243269:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698884.243273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673024:192.168.201.137@tcp:38:kworker.0 00000008:00100000:0.0:1787698884.243276:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698884.243291:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134673536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698884.243300:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698884.243305:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787698884.243306:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698884.243310:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698884.243314:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6786 00000100:00080000:0.0:1787698884.243323:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698884.243327:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698884.243331:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698884/real 1787698884] req@ffff9d85b9d0ad80 x1874531134673024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698884.243334:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698884.243350:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134673664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698884.243354:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134673024/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698884.243378:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698884.243382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698884.243386:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698884.243409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698884.243429:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698884.243439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698884/real 1787698884] req@ffff9d85b9d08a80 x1874531134673152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698884.243448:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134673152/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698884.243456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698884.243458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698884.243461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698884.243468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698884.243486:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698884.243496:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698884/real 1787698884] req@ffff9d85b9d08000 x1874531134673280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698884.243507:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134673280/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698884.243514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698884.243517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698884.243520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698884.243525:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134673408:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698884.243540:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698884.243549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698884/real 1787698884] req@ffff9d85b9d08e00 x1874531134673408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698884.243559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134673408/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698884.243567:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698884.243569:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698884.243572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134673408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698884.243578:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134673536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698884.243594:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698884.243602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698884/real 1787698884] req@ffff9d85b9d0b480 x1874531134673536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698884.243613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134673536/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698884.243621:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698884.243624:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698884.243627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134673536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698884.243631:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134673664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698884.243659:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698884.243667:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698884/real 1787698884] req@ffff9d85b9d0a680 x1874531134673664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698884.243677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134673664/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698884.243684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698884.243687:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698884.243689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134673664:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698884.819004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698885.843044:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698886.867012:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698887.891025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698888.915017:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698889.362955:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698889.362962:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698889.362965:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698889.362973:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6791 00000100:00080000:2.0:1787698889.362981:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698889.362986:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698889.363016:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134673792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698889.363042:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698889.363044:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698889.363046:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698889.363049:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6791 00000100:00080000:2.0:1787698889.363052:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698889.363055:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698889.363062:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698889.363070:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134673920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698889.363077:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698889.363080:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698889.363082:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698889.363083:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134673792:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698889.363086:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6791 00000100:00080000:2.0:1787698889.363091:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698889.363093:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698889.363096:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698889.363107:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134674048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698889.363116:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698889.363119:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698889.363121:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698889.363124:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6791 00000100:00080000:2.0:1787698889.363128:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698889.363132:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787698889.363136:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787698889.363139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698889.363155:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134674176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698889.363162:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698889.363164:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698889.363166:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698889.363169:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6791 00000100:00080000:2.0:1787698889.363172:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698889.363175:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698889/real 1787698889] req@ffff9d85babd5f80 x1874531134673792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698889.363176:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698889.363183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698889.363198:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134674304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698889.363201:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134673792/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698889.363227:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698889.363234:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698889.363237:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698889.363241:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698889.363242:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6791 00000100:00080000:2.0:1787698889.363248:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698889.363251:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698889.363256:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134673792:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698889.363262:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698889.363267:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698889.363275:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134674432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698889.363282:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673920:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698889.363282:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698889.363289:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698889.363291:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698889.363294:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6791 00000100:00080000:2.0:1787698889.363296:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698889.363298:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698889.363303:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698889.363307:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698889.363318:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134674560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698889.363319:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698889/real 1787698889] req@ffff9d85babd4700 x1874531134673920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698889.363344:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134673920/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698889.363350:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698889.363353:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698889.363357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134673920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698889.363362:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698889.363374:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698889.363382:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698889/real 1787698889] req@ffff9d85babd4e00 x1874531134674048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698889.363393:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134674048/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698889.363400:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698889.363403:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698889.363407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698889.363412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698889.363430:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698889.363439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698889/real 1787698889] req@ffff9d85babd7b80 x1874531134674176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698889.363445:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134674176/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698889.363450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698889.363452:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698889.363454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698889.363457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134674304:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698889.363467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698889.363473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698889/real 1787698889] req@ffff9d85babd4a80 x1874531134674304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698889.363479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134674304/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698889.363483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698889.363484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698889.363486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134674304:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698889.363499:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134674432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698889.363523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698889.363533:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698889/real 1787698889] req@ffff9d85babd7100 x1874531134674432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698889.363543:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134674432/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698889.363548:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698889.363549:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698889.363551:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134674432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698889.363555:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134674560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698889.363567:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698889.363572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698889/real 1787698889] req@ffff9d85babd5180 x1874531134674560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698889.363578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134674560/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698889.363582:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698889.363584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698889.363585:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134674560:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698889.939019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698890.963006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698891.986958:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698893.010997:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698894.035022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698894.482948:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698894.482954:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698894.482957:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698894.482966:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6796 00000100:00080000:2.0:1787698894.482976:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698894.482983:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698894.483017:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134674688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698894.483049:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698894.483053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698894.483055:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698894.483059:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6796 00000100:00080000:2.0:1787698894.483063:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698894.483066:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698894.483075:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698894.483088:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134674816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698894.483099:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698894.483102:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698894.483106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134674688:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698894.483106:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698894.483114:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6796 00000100:00080000:2.0:1787698894.483119:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698894.483122:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698894.483127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698894.483138:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134674944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698894.483145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698894.483147:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698894.483148:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698894.483150:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6796 00000100:00080000:2.0:1787698894.483152:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698894.483155:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698894.483159:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698894.483166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134675072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698894.483170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698894.483177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698894.483180:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698894.483181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698894.483184:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6796 00000100:00080000:2.0:1787698894.483187:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698894.483189:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698894.483192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698894.483198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134675200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698894.483203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698894.483205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698894.483206:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698894.483208:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6796 00000100:00080000:2.0:1787698894.483210:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698894.483212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698894.483215:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698894.483225:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134675328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698894.483226:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698894/real 1787698894] req@ffff9d85babd6a00 x1874531134674688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698894.483236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698894.483242:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698894.483244:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698894.483250:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6796 00000100:00080000:2.0:1787698894.483253:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698894.483255:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698894.483259:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698894.483260:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134674688/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698894.483267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134675456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698894.483293:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698894.483297:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698894.483301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134674688:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698894.483318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674816:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698894.483334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698894.483342:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698894/real 1787698894] req@ffff9d85babd6680 x1874531134674816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698894.483350:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134674816/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698894.483355:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698894.483357:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698894.483359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698894.483362:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698894.483373:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698894.483378:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698894/real 1787698894] req@ffff9d85babd4380 x1874531134674944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698894.483384:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134674944/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698894.483389:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698894.483391:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698894.483392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134674944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698894.483396:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698894.483404:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698894.483409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698894/real 1787698894] req@ffff9d85babd6300 x1874531134675072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698894.483415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134675072/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698894.483419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698894.483421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698894.483422:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698894.483425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134675200:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698894.483434:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698894.483439:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698894/real 1787698894] req@ffff9d85babd7480 x1874531134675200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698894.483445:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134675200/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698894.483449:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698894.483450:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698894.483452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134675200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698894.483455:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134675328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698894.483463:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698894.483467:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698894/real 1787698894] req@ffff9d85babd5c00 x1874531134675328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698894.483474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134675328/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698894.483478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698894.483480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698894.483481:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134675328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698894.483484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134675456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698894.483492:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698894.483496:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698894/real 1787698894] req@ffff9d85babd4000 x1874531134675456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698894.483503:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134675456/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698894.483507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698894.483509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698894.483510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134675456:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698895.059016:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698896.083013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698897.106998:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698898.131018:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698899.154981:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698899.602958:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698899.602966:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698899.602970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698899.602980:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6801 00000100:00080000:2.0:1787698899.602991:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698899.602997:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698899.603031:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134675584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698899.603055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698899.603058:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698899.603059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698899.603062:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6801 00000100:00080000:2.0:1787698899.603064:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698899.603066:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698899.603073:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698899.603084:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134675712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698899.603094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698899.603097:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698899.603099:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698899.603103:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6801 00000100:00080000:2.0:1787698899.603106:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698899.603110:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134675584:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698899.603113:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698899.603122:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698899.603133:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134675840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698899.603144:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698899.603147:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698899.603149:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698899.603152:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6801 00000100:00080000:2.0:1787698899.603156:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698899.603160:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698899.603165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698899.603176:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134675968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698899.603181:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698899.603195:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698899.603198:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698899.603200:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698899.603203:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6801 00000100:00080000:2.0:1787698899.603207:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698899.603211:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698899.603215:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698899.603226:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134676096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698899.603234:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698899.603237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698899.603239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698899.603245:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698899/real 1787698899] req@ffff9d85babd5500 x1874531134675584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698899.603245:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6801 00000100:00080000:2.0:1787698899.603251:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698899.603257:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698899.603265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698899.603278:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134676224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698899.603280:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134675584/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698899.603297:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698899.603299:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698899.603301:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698899.603309:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6801 00000100:00080000:2.0:1787698899.603316:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698899.603317:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698899.603321:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787698899.603325:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698899.603330:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134675584:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787698899.603335:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698899.603341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698899.603348:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134676352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698899.603372:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698899.603383:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698899/real 1787698899] req@ffff9d85babd6d80 x1874531134675712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698899.603390:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134675712/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698899.603395:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698899.603398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698899.603400:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698899.603430:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698899.603451:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698899.603461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698899/real 1787698899] req@ffff9d85babd7800 x1874531134675840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698899.603472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134675840/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698899.603479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698899.603482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698899.603485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698899.603491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698899.603508:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698899.603515:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698899/real 1787698899] req@ffff9d85babd4000 x1874531134675968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698899.603522:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134675968/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698899.603527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698899.603529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698899.603531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134675968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698899.603534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676096:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698899.603543:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698899.603548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698899/real 1787698899] req@ffff9d85babd5c00 x1874531134676096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698899.603561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134676096/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698899.603566:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698899.603568:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698899.603570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698899.603573:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698899.603582:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698899.603587:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698899/real 1787698899] req@ffff9d85babd7480 x1874531134676224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698899.603594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134676224/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698899.603598:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698899.603600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698899.603601:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698899.603604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698899.603612:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698899.603617:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698899/real 1787698899] req@ffff9d85babd6300 x1874531134676352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698899.603623:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134676352/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698899.603650:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698899.603653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698899.603657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676352:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698900.179041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698901.202963:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698902.227026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698903.251005:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698904.275016:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698904.722949:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698904.722959:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698904.722963:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698904.722974:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6806 00000100:00080000:2.0:1787698904.722985:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698904.722992:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698904.723027:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134676480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698904.723056:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698904.723060:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698904.723062:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698904.723066:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6806 00000100:00080000:2.0:1787698904.723070:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698904.723074:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698904.723083:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698904.723097:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134676608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698904.723107:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698904.723111:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698904.723114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698904.723116:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134676480:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698904.723120:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6806 00000100:00080000:2.0:1787698904.723127:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698904.723130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698904.723136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698904.723147:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134676736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698904.723156:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698904.723160:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698904.723162:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698904.723165:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6806 00000100:00080000:2.0:1787698904.723176:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698904.723179:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698904.723184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698904.723187:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698904.723198:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134676864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698904.723208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698904.723212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698904.723213:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698904.723217:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6806 00000100:00080000:2.0:1787698904.723221:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698904.723225:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698904.723229:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698904.723241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698904/real 1787698904] req@ffff9d85babd4380 x1874531134676480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698904.723242:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134676992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698904.723272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698904.723276:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698904.723278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698904.723280:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134676480/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698904.723285:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6806 00000100:00080000:2.0:1787698904.723292:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698904.723300:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698904.723307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698904.723308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698904.723316:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698904.723319:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134676480:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698904.723325:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134677120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698904.723340:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698904.723343:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698904.723346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134676608:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698904.723347:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698904.723356:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6806 00000100:00080000:2.0:1787698904.723359:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698904.723362:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698904.723366:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698904.723375:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134677248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698904.723381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698904.723400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698904/real 1787698904] req@ffff9d85babd6680 x1874531134676608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698904.723420:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134676608/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698904.723428:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698904.723432:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698904.723435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134676608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698904.723441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134676736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698904.723459:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698904.723468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698904/real 1787698904] req@ffff9d85babd6a00 x1874531134676736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698904.723479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134676736/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698904.723486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698904.723489:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698904.723492:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134676736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698904.723498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134676864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698904.723510:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698904.723517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698904/real 1787698904] req@ffff9d85babd5180 x1874531134676864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698904.723523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134676864/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698904.723528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698904.723529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698904.723531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134676864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698904.723534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676992:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698904.723570:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698904.723577:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698904/real 1787698904] req@ffff9d85babd7100 x1874531134676992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698904.723583:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134676992/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698904.723588:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698904.723590:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698904.723592:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134676992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698904.723595:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134677120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698904.723604:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698904.723609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698904/real 1787698904] req@ffff9d85babd4a80 x1874531134677120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698904.723617:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134677120/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698904.723624:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698904.723627:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698904.723630:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134677120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698904.723647:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134677248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698904.723662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698904.723669:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698904/real 1787698904] req@ffff9d85babd7b80 x1874531134677248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698904.723676:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134677248/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698904.723682:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698904.723684:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698904.723687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134677248:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698905.299029:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698906.322963:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698907.347022:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698908.371049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698909.395013:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698909.842980:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698909.842996:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698909.843007:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698909.843015:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6811 00000100:00080000:2.0:1787698909.843026:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698909.843031:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698909.843078:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134677376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698909.843106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698909.843108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698909.843110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698909.843113:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6811 00000100:00080000:2.0:1787698909.843115:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698909.843118:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698909.843135:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698909.843145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134677504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698909.843153:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698909.843156:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698909.843157:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134677376:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698909.843157:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698909.843164:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6811 00000100:00080000:2.0:1787698909.843169:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698909.843182:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698909.843186:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698909.843201:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134677632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698909.843208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698909.843210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698909.843212:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698909.843214:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6811 00000100:00080000:2.0:1787698909.843217:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698909.843219:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698909.843221:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698909.843238:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134677760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698909.843244:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698909.843246:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698909.843248:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698909.843250:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6811 00000100:00080000:2.0:1787698909.843252:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698909.843256:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698909.843259:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698909.843267:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134677888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698909.843273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134677504:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787698909.843290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698909.843293:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698909.843294:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698909.843298:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6811 00000100:00080000:2.0:1787698909.843302:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698909.843304:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698909.843307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698909.843316:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134678016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698909.843322:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698909.843324:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134677632:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787698909.843327:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698909.843342:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698909.843347:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6811 00000100:00080000:2.0:1787698909.843352:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698909.843356:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698909.843361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134677760:192.168.201.137@tcp:8:kworker.0 00000008:00100000:2.0:1787698909.843363:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698909.843381:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134678144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698909.843403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134677888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698909.843435:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698909.843487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678144:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698910.419025:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698911.443036:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698912.466993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698912.915301:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787698912.915365:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698912.915392:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698912.915477:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698912.915491:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698912.915500:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698912.915509:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698912.915518:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787698912.915528:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787698912.915623:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698909/real 1787698912] req@ffff9d85babd5180 x1874531134678144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698912.915651:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134678144/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698912.915664:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698912.915670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698912.915674:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698912.915682:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698909/real 1787698912] req@ffff9d85babd4a80 x1874531134677888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698912.915693:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134677888/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698912.915700:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698912.915703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698912.915706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134677888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698912.915712:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698909/real 1787698912] req@ffff9d85babd7100 x1874531134678016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698912.915722:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134678016/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698912.915735:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698912.915739:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698912.915742:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698912.915749:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698909/real 1787698912] req@ffff9d85babd4700 x1874531134677504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698912.915761:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134677504/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698912.915767:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698912.915771:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698912.915774:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134677504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698912.915780:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698909/real 1787698912] req@ffff9d85babd5f80 x1874531134677632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698912.915792:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134677632/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698912.915799:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698912.915802:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698912.915805:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134677632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698912.915812:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698909/real 1787698912] req@ffff9d85babd7b80 x1874531134677760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698912.915824:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134677760/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698912.915832:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698912.915835:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698912.915839:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134677760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698912.915846:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698909/real 1787698912] req@ffff9d85babd4e00 x1874531134677376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698912.916045:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134677376/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698912.916053:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698912.916056:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698912.916059:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134677376:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787698913.491075:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698914.515067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698914.962960:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698914.962969:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698914.962973:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698914.962983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6816 00000100:00080000:2.0:1787698914.962995:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698914.963002:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698914.963035:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134678272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698914.963061:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698914.963064:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698914.963066:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698914.963070:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6816 00000100:00080000:2.0:1787698914.963073:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698914.963077:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698914.963085:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698914.963097:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134678400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698914.963109:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698914.963115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698914.963120:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698914.963122:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134678272:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698914.963129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6816 00000100:00080000:2.0:1787698914.963139:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698914.963144:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698914.963149:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698914.963167:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134678528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698914.963177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698914.963181:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698914.963183:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698914.963187:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6816 00000100:00080000:2.0:1787698914.963194:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698914.963205:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698914.963214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698914.963227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134678656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698914.963236:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698914.963244:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698914.963246:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698914.963250:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698914.963256:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6816 00000100:00080000:2.0:1787698914.963261:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698914.963267:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698914.963271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698914.963285:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134678784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698914.963295:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698914.963298:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698914.963300:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698914.963303:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6816 00000100:00080000:2.0:1787698914.963307:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698914.963311:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698914.963315:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698914.963327:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134678912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698914.963338:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698914/real 1787698914] req@ffff9d85babd6a00 x1874531134678272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698914.963389:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698914.963395:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134678272/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698914.963396:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698914.963402:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698914.963415:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6816 00000100:00080000:2.0:1787698914.963425:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698914.963429:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698914.963430:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698914.963444:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698914.963445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698914.963461:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134678272:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698914.963470:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134679040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698914.963502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134678400:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698914.963546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698914.963557:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698914/real 1787698914] req@ffff9d85babd6680 x1874531134678400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698914.963567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134678400/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698914.963574:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698914.963576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698914.963579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134678400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698914.963586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134678528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698914.963609:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698914.963619:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698914/real 1787698914] req@ffff9d85babd4380 x1874531134678528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698914.963630:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134678528/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698914.963637:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698914.963640:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698914.963643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134678528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698914.963649:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134678656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698914.963667:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698914.963678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698914/real 1787698914] req@ffff9d85babd6300 x1874531134678656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698914.963701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134678656/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698914.963708:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698914.963711:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698914.963713:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134678656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698914.963719:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678784:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698914.963740:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698914.963750:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698914/real 1787698914] req@ffff9d85babd7480 x1874531134678784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698914.963760:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134678784/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698914.963766:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698914.963769:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698914.963772:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678784:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698914.963777:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698914.963794:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698914.963803:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698914/real 1787698914] req@ffff9d85babd5c00 x1874531134678912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698914.963813:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134678912/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698914.963825:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698914.963829:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698914.963832:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134678912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698914.963837:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698914.963901:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698914.963913:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698914/real 1787698914] req@ffff9d85babd4000 x1874531134679040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698914.963924:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134679040/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698914.963932:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698914.963936:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698914.963941:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679040:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698915.539091:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698916.563044:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698917.587088:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698918.611014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698919.635050:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698920.082929:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698920.082937:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698920.082940:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698920.082947:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6821 00000100:00080000:2.0:1787698920.082956:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698920.082961:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698920.082989:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134679168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698920.083012:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698920.083014:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698920.083015:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698920.083019:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6821 00000100:00080000:2.0:1787698920.083021:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698920.083023:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698920.083029:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698920.083038:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134679296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698920.083044:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698920.083046:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698920.083047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698920.083049:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6821 00000100:00080000:2.0:1787698920.083051:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698920.083053:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698920.083056:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698920.083059:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134679168:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698920.083065:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134679424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698920.083071:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698920.083073:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698920.083074:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698920.083076:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6821 00000100:00080000:2.0:1787698920.083078:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698920.083080:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698920.083082:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698920.083089:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134679552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698920.083095:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698920.083097:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698920.083098:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698920.083100:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6821 00000100:00080000:2.0:1787698920.083102:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698920.083105:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698920.083107:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698920.083113:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787698920.083114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134679680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698920.083120:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698920.083123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698920.083124:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698920.083127:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6821 00000100:00080000:2.0:1787698920.083129:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698920.083131:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698920.083133:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698920.083139:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134679808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698920.083144:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698920.083145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698920.083147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698920.083149:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6821 00000100:00080000:2.0:1787698920.083151:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698920.083153:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698920/real 1787698920] req@ffff9d85babd7800 x1874531134679168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698920.083156:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698920.083163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698920.083173:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134679936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698920.083182:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134679168/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698920.083194:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698920.083199:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698920.083203:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134679168:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698920.083210:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134679296:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698920.083227:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698920.083235:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698920/real 1787698920] req@ffff9d85babd6d80 x1874531134679296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698920.083243:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134679296/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698920.083249:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698920.083251:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698920.083258:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134679296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698920.083275:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134679424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698920.083312:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698920.083319:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698920/real 1787698920] req@ffff9d85babd5500 x1874531134679424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698920.083327:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134679424/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698920.083332:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698920.083334:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698920.083336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134679424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698920.083339:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134679552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698920.083349:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698920.083354:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698920/real 1787698920] req@ffff9d85babd4000 x1874531134679552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698920.083361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134679552/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698920.083365:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698920.083367:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698920.083369:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134679552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698920.083372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679680:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698920.083380:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698920.083385:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698920/real 1787698920] req@ffff9d85babd5c00 x1874531134679680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698920.083392:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134679680/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698920.083399:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698920.083402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698920.083405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698920.083410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698920.083428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698920.083438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698920/real 1787698920] req@ffff9d85babd7480 x1874531134679808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698920.083447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134679808/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698920.083456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698920.083458:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698920.083460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698920.083463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698920.083475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698920.083481:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698920/real 1787698920] req@ffff9d85babd6300 x1874531134679936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698920.083487:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134679936/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698920.083492:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698920.083493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698920.083494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134679936:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698920.659034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698921.683018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698922.707010:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698923.731010:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698924.755069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698925.202976:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698925.202984:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698925.202987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698925.202994:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6826 00000100:00080000:2.0:1787698925.203003:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698925.203008:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698925.203036:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134680064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698925.203056:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698925.203059:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698925.203060:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698925.203063:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6826 00000100:00080000:2.0:1787698925.203065:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698925.203067:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698925.203073:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698925.203080:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134680192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698925.203087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698925.203089:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698925.203091:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698925.203093:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6826 00000100:00080000:2.0:1787698925.203094:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698925.203097:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698925.203099:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698925.203105:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134680320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698925.203112:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698925.203113:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134680064:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698925.203114:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698925.203117:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698925.203123:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6826 00000100:00080000:2.0:1787698925.203125:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698925.203127:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698925.203130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698925.203137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134680448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698925.203142:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698925.203144:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698925.203146:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698925.203147:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6826 00000100:00080000:2.0:1787698925.203150:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698925.203152:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698925.203155:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698925.203160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134680576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698925.203165:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698925.203167:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698925.203168:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698925.203170:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6826 00000100:00080000:2.0:1787698925.203173:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698925.203177:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698925.203177:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698925.203183:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698925.203206:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134680704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698925.203216:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698925.203218:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698925.203219:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698925.203222:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6826 00000100:00080000:2.0:1787698925.203224:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698925.203227:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698925.203231:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698925.203238:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698925/real 1787698925] req@ffff9d85babd4380 x1874531134680064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698925.203241:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134680832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698925.203269:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134680064/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698925.203280:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698925.203283:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698925.203287:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134680064:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698925.203295:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134680192:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698925.203315:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698925.203326:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698925/real 1787698925] req@ffff9d85babd6680 x1874531134680192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698925.203338:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134680192/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698925.203346:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698925.203349:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698925.203352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134680192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698925.203372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134680320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698925.203389:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698925.203398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698925/real 1787698925] req@ffff9d85babd6a00 x1874531134680320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698925.203409:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134680320/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698925.203417:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698925.203419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698925.203423:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134680320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698925.203428:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134680448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698925.203442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698925.203452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698925/real 1787698925] req@ffff9d85babd4e00 x1874531134680448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698925.203462:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134680448/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698925.203470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698925.203473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698925.203476:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134680448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698925.203481:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134680576:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698925.203496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698925.203511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698925/real 1787698925] req@ffff9d85babd7b80 x1874531134680576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698925.203520:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134680576/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698925.203527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698925.203529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698925.203532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134680576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698925.203536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134680704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698925.203551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698925.203558:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698925/real 1787698925] req@ffff9d85babd5f80 x1874531134680704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698925.203567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134680704/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698925.203579:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698925.203581:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698925.203584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134680704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698925.203588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134680832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698925.203602:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698925.203610:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698925/real 1787698925] req@ffff9d85babd4700 x1874531134680832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698925.203620:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134680832/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698925.203626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698925.203629:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698925.203631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134680832:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698925.779010:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698926.804355:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698927.827007:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698928.851041:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698929.875013:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698930.323028:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698930.323039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698930.323044:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698930.323054:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6832 00000100:00080000:1.0:1787698930.323066:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698930.323074:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698930.323147:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134680960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698930.323178:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698930.323181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698930.323183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698930.323186:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6832 00000100:00080000:1.0:1787698930.323189:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698930.323192:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698930.323202:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698930.323210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134681088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698930.323217:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698930.323219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698930.323220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698930.323223:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6832 00000100:00080000:1.0:1787698930.323225:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698930.323227:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698930.323230:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698930.323237:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134681216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698930.323239:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134680960:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698930.323243:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698930.323248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698930.323249:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698930.323251:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6832 00000100:00080000:1.0:1787698930.323253:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698930.323254:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698930.323258:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698930.323265:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134681344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698930.323270:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698930.323272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698930.323274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698930.323276:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6832 00000100:00080000:1.0:1787698930.323278:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698930.323280:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698930.323282:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698930.323288:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134681472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698930.323293:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698930.323296:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698930.323297:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698930.323299:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6832 00000100:00080000:1.0:1787698930.323309:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698930.323312:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698930.323314:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698930.323323:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134681600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698930.323325:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698930.323332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698930.323335:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698930.323337:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698930.323339:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6832 00000100:00080000:1.0:1787698930.323340:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698930.323342:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698930.323345:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698930.323352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134681728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698930.323386:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698930/real 1787698930] req@ffff9d85b9d2fb80 x1874531134680960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698930.323406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134680960/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698930.323423:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698930.323428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698930.323433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134680960:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698930.323441:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698930.323465:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698930.323477:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698930/real 1787698930] req@ffff9d85b9d2f800 x1874531134681088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698930.323489:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134681088/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698930.323497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698930.323500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698930.323503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698930.323532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698930.323553:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698930.323563:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698930/real 1787698930] req@ffff9d85b9d2ca80 x1874531134681216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698930.323574:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134681216/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698930.323581:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698930.323585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698930.323588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698930.323594:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698930.323612:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698930.323621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698930/real 1787698930] req@ffff9d85b9d2f480 x1874531134681344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698930.323632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134681344/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698930.323640:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698930.323643:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698930.323647:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698930.323652:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134681472:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698930.323671:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698930.323681:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698930/real 1787698930] req@ffff9d85b9d2e300 x1874531134681472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698930.323693:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134681472/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698930.323701:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698930.323704:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698930.323707:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134681472:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698930.323713:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134681600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698930.323745:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698930.323755:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698930/real 1787698930] req@ffff9d85b9d2c000 x1874531134681600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698930.323767:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134681600/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698930.323775:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698930.323778:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698930.323781:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134681600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698930.323786:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134681728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698930.323805:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698930.323814:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698930/real 1787698930] req@ffff9d85b9d2ce00 x1874531134681728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698930.323825:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134681728/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698930.323833:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698930.323836:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698930.323839:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134681728:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698930.899015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698931.923113:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698932.947039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698933.971008:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698934.995748:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698935.443067:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698935.443077:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698935.443081:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698935.443091:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6837 00000100:00080000:0.0:1787698935.443101:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698935.443107:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698935.443151:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134681856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698935.443189:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698935.443192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698935.443194:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698935.443198:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6837 00000100:00080000:0.0:1787698935.443201:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698935.443205:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698935.443214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698935.443225:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134681984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698935.443234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698935.443237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698935.443238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698935.443241:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6837 00000100:00080000:0.0:1787698935.443245:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698935.443249:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698935.443253:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698935.443261:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134682112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698935.443269:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698935.443272:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698935.443273:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698935.443276:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6837 00000100:00080000:0.0:1787698935.443279:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698935.443283:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698935.443287:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698935.443296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134682240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698935.443304:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698935.443307:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698935.443308:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698935.443312:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6837 00000100:00080000:0.0:1787698935.443314:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698935.443318:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698935.443322:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698935.443340:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134682368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698935.443348:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698935.443351:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698935.443353:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698935.443357:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6837 00000100:00080000:0.0:1787698935.443360:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698935.443363:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698935.443367:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698935.443376:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134682496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698935.443389:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698935.443392:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698935.443394:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698935.443397:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6837 00000100:00080000:0.0:1787698935.443400:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698935.443403:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787698935.443407:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698935.443416:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134682624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698935.443556:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134681856:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698935.443636:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698935.443683:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698935/real 1787698935] req@ffff9d85b9d09c00 x1874531134681856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698935.443700:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134681856/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698935.443711:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698935.443714:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698935.443718:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134681856:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698935.443724:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681984:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698935.443741:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698935.443749:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698935/real 1787698935] req@ffff9d85b9d08700 x1874531134681984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698935.443756:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134681984/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698935.443765:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698935.443768:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698935.443770:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134681984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698935.443772:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134682112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698935.443782:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698935.443787:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698935/real 1787698935] req@ffff9d85b9d09500 x1874531134682112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698935.443793:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134682112/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698935.443798:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698935.443799:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698935.443801:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134682112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698935.443804:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134682240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698935.443812:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698935.443817:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698935/real 1787698935] req@ffff9d85b9d09180 x1874531134682240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698935.443823:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134682240/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698935.443837:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698935.443840:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698935.443843:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134682240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698935.443848:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134682368:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698935.443891:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698935.443898:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698935/real 1787698935] req@ffff9d85b9d0a300 x1874531134682368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698935.443905:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134682368/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698935.443910:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698935.443912:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698935.443914:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134682368:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698935.443917:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134682496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698935.443926:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698935.443931:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698935/real 1787698935] req@ffff9d85b9d09f80 x1874531134682496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698935.443938:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134682496/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698935.443942:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698935.443943:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698935.443945:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134682496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698935.443947:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134682624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698935.443956:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698935.443961:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698935/real 1787698935] req@ffff9d85b9d0b800 x1874531134682624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698935.443967:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134682624/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698935.443971:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698935.443973:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698935.443974:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134682624:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698936.019515:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698937.043069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698938.067040:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698939.091036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698940.114985:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698940.563318:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698940.563328:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698940.563369:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698940.563404:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6842 00000100:00080000:2.0:1787698940.563490:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698940.563498:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698940.563531:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134682752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698940.563556:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698940.563559:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698940.563561:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698940.563565:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6842 00000100:00080000:2.0:1787698940.563568:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698940.563573:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698940.563583:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698940.563598:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134682752:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698940.563598:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134682880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698940.563615:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698940.563618:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698940.563619:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698940.563624:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6842 00000100:00080000:2.0:1787698940.563627:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698940.563631:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698940.563636:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698940.563646:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134683008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698940.563655:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698940.563658:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698940.563661:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698940.563665:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698940.563669:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6842 00000100:00080000:2.0:1787698940.563673:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698940.563677:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698940.563681:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698940.563693:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134683136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698940.563702:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698940.563705:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698940.563707:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698940.563711:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6842 00000100:00080000:2.0:1787698940.563714:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698940.563716:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698940/real 1787698940] req@ffff9d85babd7100 x1874531134682752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698940.563722:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698940.563730:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698940.563741:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134682752/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698940.563744:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134683264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698940.563756:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698940.563761:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698940.563762:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698940.563769:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134682752:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698940.563769:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698940.563773:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698940.563780:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6842 00000100:00080000:2.0:1787698940.563784:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698940.563788:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698940.563792:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134682880:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787698940.563793:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698940.563810:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134683392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698940.563817:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698940.563827:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698940.563829:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698940/real 1787698940] req@ffff9d85babd4a80 x1874531134682880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698940.563832:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698940.563837:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698940.563845:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6842 00000100:00100000:1.0:1787698940.563846:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134682880/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698940.563888:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698940.563892:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698940.563896:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134682880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698940.563903:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698940.563924:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698940.563926:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698940.563943:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698940.563949:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698940/real 1787698940] req@ffff9d85babd5180 x1874531134683008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698940.563950:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698940.563963:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134683008/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698940.563970:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134683520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698940.563971:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698940.563977:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698940.563983:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698940.563994:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698940.564015:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698940.564026:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698940/real 1787698940] req@ffff9d85babd4700 x1874531134683136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698940.564037:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134683136/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698940.564045:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698940.564048:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698940.564052:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698940.564057:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134683264:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698940.564074:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698940.564084:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698940/real 1787698940] req@ffff9d85babd5f80 x1874531134683264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698940.564095:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134683264/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698940.564103:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698940.564105:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698940.564109:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134683264:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698940.564123:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134683392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698940.564136:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698940.564143:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698940/real 1787698940] req@ffff9d85babd7b80 x1874531134683392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698940.564149:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134683392/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698940.564154:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698940.564156:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698940.564158:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134683392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698940.564161:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134683520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698940.564169:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698940.564174:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698940/real 1787698940] req@ffff9d85babd4e00 x1874531134683520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698940.564180:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134683520/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698940.564185:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698940.564186:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698940.564188:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134683520:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698941.139014:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698942.163066:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698943.187380:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698944.211043:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698945.235911:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698945.682987:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698945.682999:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698945.683004:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698945.683017:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6847 00000100:00080000:2.0:1787698945.683031:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698945.683039:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698945.683086:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134683648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698945.683123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698945.683127:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698945.683129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698945.683134:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6847 00000100:00080000:2.0:1787698945.683137:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698945.683140:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698945.683151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698945.683165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134683776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698945.683175:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698945.683179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698945.683181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698945.683184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6847 00000100:00080000:2.0:1787698945.683187:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698945.683190:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698945.683195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698945.683204:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134683904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698945.683213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698945.683216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698945.683218:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698945.683221:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6847 00000100:00080000:2.0:1787698945.683224:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698945.683228:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698945.683232:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698945.683242:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134684032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698945.683251:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698945.683254:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698945.683256:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698945.683259:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6847 00000100:00080000:2.0:1787698945.683262:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698945.683266:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698945.683270:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698945.683279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134684160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698945.683288:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698945.683291:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698945.683293:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698945.683296:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6847 00000100:00080000:2.0:1787698945.683299:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698945.683303:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698945.683307:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698945.683317:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134684288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698945.683325:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698945.683328:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698945.683330:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698945.683333:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6847 00000100:00080000:2.0:1787698945.683336:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698945.683339:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698945.683343:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698945.683352:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134684416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698945.683449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134683648:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698945.683523:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698945.683575:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698945/real 1787698945] req@ffff9d85babd6a00 x1874531134683648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698945.683597:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134683648/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698945.683608:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698945.683613:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698945.683617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134683648:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698945.683624:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683776:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698945.683644:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698945.683656:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698945/real 1787698945] req@ffff9d85babd6680 x1874531134683776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698945.683668:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134683776/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698945.683683:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698945.683686:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698945.683689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698945.683694:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698945.683712:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698945.683722:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698945/real 1787698945] req@ffff9d85babd4380 x1874531134683904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698945.683732:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134683904/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698945.683739:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698945.683742:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698945.683745:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134683904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698945.683751:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698945.683769:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698945.683779:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698945/real 1787698945] req@ffff9d85babd6300 x1874531134684032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698945.683789:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134684032/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698945.683797:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698945.683800:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698945.683803:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698945.683808:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134684160:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698945.683826:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698945.683837:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698945/real 1787698945] req@ffff9d85babd7480 x1874531134684160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698945.683883:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134684160/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698945.683891:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698945.683894:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698945.683897:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134684160:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698945.683903:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134684288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698945.683922:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698945.683932:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698945/real 1787698945] req@ffff9d85babd5c00 x1874531134684288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698945.683944:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134684288/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698945.683952:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698945.683955:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698945.683957:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134684288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698945.683963:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134684416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698945.683981:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698945.683992:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698945/real 1787698945] req@ffff9d85babd4000 x1874531134684416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698945.684003:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134684416/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698945.684012:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698945.684014:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698945.684017:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134684416:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698946.259136:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698947.283040:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698948.307252:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698949.331092:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698950.355057:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698950.803007:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698950.803021:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698950.803026:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698950.803039:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6852 00000100:00080000:2.0:1787698950.803053:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698950.803061:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698950.803110:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134684544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698950.803150:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698950.803154:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698950.803156:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698950.803160:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6852 00000100:00080000:2.0:1787698950.803164:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698950.803167:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698950.803177:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698950.803189:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134684672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698950.803198:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698950.803201:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698950.803203:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698950.803206:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6852 00000100:00080000:2.0:1787698950.803209:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698950.803212:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698950.803217:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698950.803225:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134684800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698950.803234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698950.803237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698950.803238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698950.803251:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6852 00000100:00080000:2.0:1787698950.803254:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698950.803258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698950.803263:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698950.803274:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134684928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698950.803283:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698950.803286:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698950.803288:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698950.803291:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6852 00000100:00080000:2.0:1787698950.803295:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698950.803298:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698950.803303:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698950.803313:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134685056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698950.803322:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698950.803325:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698950.803327:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698950.803330:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6852 00000100:00080000:2.0:1787698950.803333:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698950.803338:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698950.803343:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698950.803352:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134685184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698950.803360:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698950.803363:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698950.803364:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698950.803367:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6852 00000100:00080000:2.0:1787698950.803370:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698950.803374:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698950.803378:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698950.803388:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134685312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698950.803508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134684544:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787698950.803586:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698950.803648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698950/real 1787698950] req@ffff9d85babd5500 x1874531134684544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698950.803669:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134684544/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698950.803681:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698950.803685:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698950.803689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134684544:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698950.803697:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684672:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698950.803719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698950.803731:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698950/real 1787698950] req@ffff9d85babd6d80 x1874531134684672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698950.803742:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134684672/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698950.803928:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698950.803933:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698950.803936:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698950.803943:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698950.803963:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698950.803974:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698950/real 1787698950] req@ffff9d85babd7800 x1874531134684800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698950.803984:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134684800/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698950.803992:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698950.803995:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698950.803998:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698950.804004:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698950.804020:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698950.804029:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698950/real 1787698950] req@ffff9d85babd4000 x1874531134684928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698950.804039:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134684928/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698950.804046:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698950.804049:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698950.804053:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134684928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698950.804058:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685056:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698950.804074:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698950.804083:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698950/real 1787698950] req@ffff9d85babd5c00 x1874531134685056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698950.804093:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134685056/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698950.804100:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698950.804103:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698950.804106:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698950.804112:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698950.804126:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698950.804135:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698950/real 1787698950] req@ffff9d85babd7480 x1874531134685184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698950.804145:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134685184/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698950.804152:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698950.804155:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698950.804158:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698950.804163:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698950.804176:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698950.804185:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698950/real 1787698950] req@ffff9d85babd6300 x1874531134685312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698950.804195:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134685312/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698950.804202:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698950.804204:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698950.804207:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685312:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698951.379024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698952.403156:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698953.426999:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698954.451064:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698955.475025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698955.923436:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698955.923449:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698955.923455:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698955.923469:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6857 00000100:00080000:3.0:1787698955.923495:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698955.923505:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698955.923552:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134685440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698955.923581:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698955.923585:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698955.923587:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698955.923591:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6857 00000100:00080000:3.0:1787698955.923594:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698955.923598:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698955.923609:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698955.923620:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134685568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698955.923631:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698955.923634:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698955.923641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134685440:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698955.923645:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698955.923651:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6857 00000100:00080000:3.0:1787698955.923653:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698955.923657:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698955.923662:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698955.923699:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134685696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698955.923709:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698955.923712:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698955.923713:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698955.923717:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6857 00000100:00080000:3.0:1787698955.923728:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698955.923731:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698955.923735:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698955.923739:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698955.923751:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134685824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698955.923761:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698955.923765:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698955.923767:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698955.923771:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6857 00000100:00080000:3.0:1787698955.923774:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698955.923778:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698955.923782:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698955.923800:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134685952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698955.923809:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698955.923811:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698955/real 1787698955] req@ffff9d85c0fb1c00 x1874531134685440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698955.923814:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698955.923818:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698955.923825:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6857 00000100:00080000:3.0:1787698955.923830:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698955.923835:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698955.923839:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698955.923840:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134685440/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698955.923877:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134686080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698955.923884:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698955.923889:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698955.923893:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787698955.923894:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698955.923898:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698955.923902:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134685440:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787698955.923904:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6857 00000100:00080000:3.0:1787698955.923910:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698955.923915:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698955.923919:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698955.923930:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134686208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698955.923937:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134685568:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698955.923965:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698955.923979:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698955/real 1787698955] req@ffff9d85c0fb0a80 x1874531134685568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698955.923990:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134685568/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698955.924004:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698955.924007:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698955.924011:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134685568:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698955.924017:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134685696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698955.924035:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698955.924046:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698955/real 1787698955] req@ffff9d85c0fb3b80 x1874531134685696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698955.924057:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134685696/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698955.924064:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698955.924066:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698955.924069:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134685696:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698955.924074:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134685824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698955.924092:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698955.924102:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698955/real 1787698955] req@ffff9d85c0fb0380 x1874531134685824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698955.924111:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134685824/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698955.924119:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698955.924121:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698955.924125:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134685824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698955.924130:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698955.924148:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698955.924158:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698955/real 1787698955] req@ffff9d85c0fb2d80 x1874531134685952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698955.924168:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134685952/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698955.924174:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698955.924177:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698955.924180:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134685952:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698955.924185:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698955.924202:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698955.924227:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698955/real 1787698955] req@ffff9d85c0fb3800 x1874531134686080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698955.924239:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134686080/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698955.924247:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698955.924249:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698955.924252:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698955.924269:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698955.924288:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698955.924300:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698955/real 1787698955] req@ffff9d85c0fb0000 x1874531134686208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698955.924311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134686208/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698955.924319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698955.924321:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698955.924324:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686208:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698956.499051:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698957.523053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698958.547085:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698959.571055:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698960.595069:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698961.042952:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698961.042963:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698961.042967:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698961.042978:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6862 00000100:00080000:2.0:1787698961.042990:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698961.042998:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698961.043052:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134686336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698961.043079:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698961.043082:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698961.043083:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698961.043087:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6862 00000100:00080000:2.0:1787698961.043090:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698961.043093:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698961.043103:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698961.043114:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134686464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698961.043120:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698961.043123:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698961.043125:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698961.043127:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6862 00000100:00080000:2.0:1787698961.043129:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698961.043132:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698961.043136:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698961.043140:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134686336:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787698961.043144:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134686592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698961.043155:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698961.043157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698961.043158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698961.043161:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6862 00000100:00080000:2.0:1787698961.043163:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698961.043165:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698961.043167:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698961.043175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134686720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698961.043184:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698961.043186:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698961.043187:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698961.043189:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6862 00000100:00080000:2.0:1787698961.043191:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698961.043194:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698961.043197:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698961.043204:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134686848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698961.043209:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698961.043211:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698961.043213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698961.043216:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6862 00000100:00080000:2.0:1787698961.043218:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698961.043220:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698961.043222:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698961.043231:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134686976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698961.043232:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698961.043240:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698961.043244:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698961.043246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698961.043250:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6862 00000100:00080000:2.0:1787698961.043254:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698961.043259:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698961.043264:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698961.043276:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134687104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698961.043307:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698961/real 1787698961] req@ffff9d85babd4380 x1874531134686336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698961.043337:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134686336/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698961.043348:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698961.043352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698961.043356:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134686336:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698961.043364:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134686464:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698961.043386:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698961.043398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698961/real 1787698961] req@ffff9d85babd6680 x1874531134686464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698961.043410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134686464/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698961.043423:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698961.043427:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698961.043430:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134686464:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698961.043455:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134686592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698961.043473:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698961.043483:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698961/real 1787698961] req@ffff9d85babd6a00 x1874531134686592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698961.043493:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134686592/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698961.043501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698961.043504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698961.043507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134686592:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698961.043513:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134686720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698961.043526:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698961.043535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698961/real 1787698961] req@ffff9d85babd4e00 x1874531134686720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698961.043545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134686720/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698961.043552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698961.043554:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698961.043558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134686720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698961.043563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686848:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698961.043577:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698961.043599:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698961/real 1787698961] req@ffff9d85babd7b80 x1874531134686848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698961.043610:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134686848/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698961.043617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698961.043619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698961.043622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686848:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698961.043629:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698961.043645:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698961.043654:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698961/real 1787698961] req@ffff9d85babd5f80 x1874531134686976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698961.043664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134686976/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698961.043671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698961.043674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698961.043677:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134686976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698961.043683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134687104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698961.043697:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698961.043706:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698961/real 1787698961] req@ffff9d85babd4700 x1874531134687104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698961.043716:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134687104/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698961.043723:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698961.043726:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698961.043729:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134687104:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698961.619135:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698962.643008:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698963.667088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698964.691080:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698965.715087:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698966.162981:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698966.162992:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698966.162996:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698966.163006:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6867 00000100:00080000:2.0:1787698966.163017:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698966.163025:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698966.163079:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134687232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698966.163107:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698966.163111:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698966.163113:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698966.163118:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6867 00000100:00080000:2.0:1787698966.163122:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698966.163126:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698966.163139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698966.163151:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134687360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698966.163158:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698966.163160:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698966.163161:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698966.163165:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6867 00000100:00080000:2.0:1787698966.163169:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698966.163172:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698966.163176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698966.163186:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134687488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698966.163187:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134687232:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698966.163199:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698966.163206:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698966.163207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698966.163215:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6867 00000100:00080000:2.0:1787698966.163218:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698966.163220:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698966.163224:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698966.163232:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134687616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698966.163242:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698966.163245:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698966.163247:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698966.163250:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6867 00000100:00080000:2.0:1787698966.163253:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698966.163255:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698966.163258:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698966.163264:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134687744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698966.163270:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698966.163276:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698966.163277:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698966.163281:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698966.163285:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6867 00000100:00080000:2.0:1787698966.163288:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698966.163292:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698966.163295:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698966.163307:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134687872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698966.163313:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698966.163316:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698966.163317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698966.163319:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6867 00000100:00080000:2.0:1787698966.163323:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698966.163326:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698966.163329:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698966.163333:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698966/real 1787698966] req@ffff9d85babd5180 x1874531134687232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698966.163342:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134688000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698966.163402:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134687232/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698966.163415:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698966.163419:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698966.163425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134687232:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787698966.163434:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134687360:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698966.163460:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698966.163472:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698966/real 1787698966] req@ffff9d85babd4a80 x1874531134687360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698966.163484:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134687360/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698966.163526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698966.163530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698966.163533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134687360:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698966.163553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134687488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698966.163569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698966.163577:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698966/real 1787698966] req@ffff9d85babd7100 x1874531134687488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698966.163584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134687488/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698966.163589:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698966.163599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698966.163601:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134687488:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698966.163605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134687616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698966.163616:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698966.163625:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698966/real 1787698966] req@ffff9d85babd4700 x1874531134687616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698966.163635:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134687616/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698966.163642:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698966.163645:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698966.163649:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134687616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698966.163655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134687744:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787698966.163674:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698966.163685:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698966/real 1787698966] req@ffff9d85babd5f80 x1874531134687744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698966.163696:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134687744/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698966.163704:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698966.163707:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698966.163710:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134687744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698966.163716:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134687872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698966.163733:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698966.163741:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698966/real 1787698966] req@ffff9d85babd7b80 x1874531134687872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698966.163748:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134687872/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698966.163752:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698966.163754:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698966.163756:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134687872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787698966.163760:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787698966.163769:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698966.163775:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698966/real 1787698966] req@ffff9d85babd4e00 x1874531134688000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698966.163781:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134688000/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698966.163786:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698966.163788:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698966.163789:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688000:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698966.739075:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698967.763046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698968.787013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698969.811032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698970.835009:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698971.283091:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698971.283100:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698971.283104:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698971.283115:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6873 00000100:00080000:1.0:1787698971.283133:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698971.283140:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698971.283177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134688128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698971.283205:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698971.283209:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698971.283211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698971.283215:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6873 00000100:00080000:1.0:1787698971.283218:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698971.283222:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698971.283231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698971.283244:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134688256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698971.283256:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134688128:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698971.283258:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698971.283266:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698971.283268:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698971.283272:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6873 00000100:00080000:1.0:1787698971.283275:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698971.283279:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698971.283285:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698971.283297:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134688384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698971.283306:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698971.283309:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698971.283311:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698971.283314:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6873 00000100:00080000:1.0:1787698971.283318:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698971.283323:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787698971.283328:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787698971.283330:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698971.283344:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134688512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698971.283353:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698971.283357:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698971.283358:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698971.283362:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6873 00000100:00080000:1.0:1787698971.283366:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698971.283373:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698971.283374:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698971/real 1787698971] req@ffff9d85b9d2dc00 x1874531134688128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698971.283379:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698971.283397:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134688128/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698971.283405:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134688640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698971.283424:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698971.283429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698971.283434:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134688128:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698971.283449:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698971.283452:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698971.283454:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698971.283459:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6873 00000100:00100000:0.0:1787698971.283463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134688256:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787698971.283469:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698971.283477:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698971.283482:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698971.283497:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134688768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698971.283498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698971.283511:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698971.283516:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698971.283517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698971/real 1787698971] req@ffff9d85b9d2e680 x1874531134688256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698971.283519:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698971.283526:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6873 00000100:00080000:1.0:1787698971.283538:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698971.283542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134688256/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698971.283545:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698971.283552:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698971.283559:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698971.283566:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787698971.283567:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134688896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698971.283575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134688256:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698971.283589:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134688384:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698971.283615:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698971.283627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698971/real 1787698971] req@ffff9d85b9d2d500 x1874531134688384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698971.283637:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134688384/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698971.283646:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698971.283649:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698971.283653:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134688384:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698971.283659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134688512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698971.283679:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698971.283690:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698971/real 1787698971] req@ffff9d85b9d2ed80 x1874531134688512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698971.283701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134688512/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698971.283715:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698971.283718:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698971.283721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134688512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698971.283726:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688640:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698971.283744:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698971.283754:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698971/real 1787698971] req@ffff9d85b9d2c700 x1874531134688640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698971.283765:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134688640/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698971.283772:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698971.283775:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698971.283777:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698971.283793:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698971.283810:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698971.283819:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698971/real 1787698971] req@ffff9d85b9d2c380 x1874531134688768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698971.283829:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134688768/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698971.283836:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698971.283839:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698971.283842:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698971.283848:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698971.284084:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698971.284096:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698971/real 1787698971] req@ffff9d85b9d2dc00 x1874531134688896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698971.284108:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134688896/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698971.284116:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698971.284119:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698971.284121:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134688896:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698971.859035:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698972.882998:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698973.907108:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698974.931013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698975.954998:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698976.402956:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698976.402968:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698976.402973:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698976.402983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6878 00000100:00080000:1.0:1787698976.402993:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698976.403001:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698976.403041:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134689024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698976.403074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698976.403077:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698976.403078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698976.403081:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6878 00000100:00080000:1.0:1787698976.403085:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698976.403087:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698976.403095:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698976.403105:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134689152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698976.403112:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698976.403114:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698976.403115:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698976.403118:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6878 00000100:00080000:1.0:1787698976.403120:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698976.403122:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698976.403129:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134689024:192.168.201.137@tcp:250:kworker.0 00000008:00100000:1.0:1787698976.403131:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698976.403142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134689280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698976.403148:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698976.403150:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698976.403152:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698976.403154:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6878 00000100:00080000:1.0:1787698976.403156:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698976.403158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698976.403161:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698976.403168:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134689408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698976.403174:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698976.403176:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698976.403177:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698976.403179:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6878 00000100:00080000:1.0:1787698976.403189:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698976.403191:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698976.403194:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698976.403203:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134689536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698976.403213:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698976.403216:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698976.403218:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698976.403222:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6878 00000100:00080000:1.0:1787698976.403225:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698976.403230:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698976.403234:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134689152:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787698976.403237:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698976.403253:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134689664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698976.403262:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698976.403266:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698976.403268:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698976.403272:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6878 00000100:00080000:1.0:1787698976.403275:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698976.403280:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787698976.403284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134689280:192.168.201.137@tcp:8:kworker.0 00000008:00100000:1.0:1787698976.403287:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698976.403303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134689792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698976.403315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134689408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698976.403355:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134689536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698976.403379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134689664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698976.403402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134689792:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698976.979031:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698978.003024:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698979.027077:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698979.475136:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:3.0:1787698979.475180:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698979.475195:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698979.475231:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698979.475240:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698979.475251:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698979.475261:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698979.475281:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698976/real 1787698979] req@ffff9d85b9d2e300 x1874531134689280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698979.475281:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787698979.475294:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787698979.475317:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134689280/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698979.475327:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698979.475331:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698979.475334:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134689280:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698979.475341:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698976/real 1787698979] req@ffff9d85b9d2f480 x1874531134689408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698979.475351:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134689408/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698979.475357:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698979.475360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698979.475362:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134689408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698979.475373:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698976/real 1787698979] req@ffff9d85b9d2ca80 x1874531134689536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698979.475386:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134689536/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698979.475392:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698979.475395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698979.475397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134689536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698979.475403:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698976/real 1787698979] req@ffff9d85b9d2f800 x1874531134689664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698979.475413:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134689664/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698979.475420:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698979.475423:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698979.475426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134689664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698979.475431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698976/real 1787698979] req@ffff9d85b9d2ce00 x1874531134689024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698979.475442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134689024/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698979.475448:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698979.475451:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698979.475454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134689024:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787698979.475460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698976/real 1787698979] req@ffff9d85b9d2c000 x1874531134689152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698979.475471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134689152/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698979.475478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698979.475481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698979.475484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134689152:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698979.475515:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698976/real 1787698979] req@ffff9d85b9d2fb80 x1874531134689792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698979.475528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134689792/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698979.475536:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698979.475540:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698979.475543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134689792:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698980.051041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698981.075020:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698981.523027:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698981.523035:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698981.523038:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698981.523047:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6883 00000100:00080000:3.0:1787698981.523055:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698981.523061:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698981.523093:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134689920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698981.523122:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698981.523126:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698981.523128:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698981.523132:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6883 00000100:00080000:3.0:1787698981.523134:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698981.523137:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698981.523145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698981.523157:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134690048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698981.523166:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698981.523169:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698981.523170:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698981.523174:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6883 00000100:00080000:3.0:1787698981.523177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698981.523180:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698981.523186:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698981.523195:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134690176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698981.523203:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698981.523205:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698981.523207:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698981.523211:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6883 00000100:00080000:3.0:1787698981.523214:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698981.523218:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698981.523222:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698981.523233:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134690304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698981.523253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698981.523256:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698981.523258:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698981.523262:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6883 00000100:00080000:3.0:1787698981.523265:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698981.523269:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698981.523273:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698981.523281:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134689920:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698981.523286:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134690432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698981.523297:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698981.523301:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698981.523303:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698981.523306:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6883 00000100:00080000:3.0:1787698981.523310:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698981.523314:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698981.523318:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698981.523331:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134690560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698981.523348:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698981.523390:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698981/real 1787698981] req@ffff9d85c0fb0e00 x1874531134689920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698981.523395:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698981.523402:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698981.523406:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698981.523412:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6883 00000100:00080000:3.0:1787698981.523416:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698981.523417:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134689920/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698981.523424:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698981.523434:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698981.523438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698981.523445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698981.523449:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134689920:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698981.523451:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134690688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698981.523461:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690048:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698981.523484:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698981.523498:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698981/real 1787698981] req@ffff9d85c0fb2680 x1874531134690048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698981.523509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134690048/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698981.523523:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698981.523525:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698981.523528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690048:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698981.523533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698981.523550:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698981.523558:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698981/real 1787698981] req@ffff9d85c0fb2a00 x1874531134690176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698981.523569:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134690176/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698981.523576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698981.523579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698981.523582:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690176:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698981.523587:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698981.523601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698981.523609:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698981/real 1787698981] req@ffff9d85c0fb0000 x1874531134690304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698981.523619:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134690304/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698981.523625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698981.523628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698981.523632:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698981.523664:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134690432:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698981.523681:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698981.523691:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698981/real 1787698981] req@ffff9d85c0fb3800 x1874531134690432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698981.523701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134690432/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698981.523708:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698981.523710:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698981.523714:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134690432:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698981.523719:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134690560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698981.523732:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698981.523740:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698981/real 1787698981] req@ffff9d85c0fb2d80 x1874531134690560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698981.523751:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134690560/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698981.523758:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698981.523761:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698981.523764:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134690560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698981.523769:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134690688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698981.523781:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698981.523790:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698981/real 1787698981] req@ffff9d85c0fb0380 x1874531134690688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698981.523800:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134690688/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698981.523808:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698981.523811:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698981.523814:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134690688:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698982.098976:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698983.123011:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698984.147030:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698985.170996:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698986.195041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698986.643107:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698986.643118:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698986.643123:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698986.643132:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6888 00000100:00080000:2.0:1787698986.643144:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698986.643165:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787698986.643203:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134690816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698986.643236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698986.643240:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698986.643242:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698986.643246:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6888 00000100:00080000:2.0:1787698986.643249:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698986.643253:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698986.643262:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698986.643280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134690944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698986.643288:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698986.643291:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698986.643293:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698986.643296:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6888 00000100:00080000:2.0:1787698986.643300:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698986.643305:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698986.643310:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698986.643320:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134691072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698986.643328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134690816:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787698986.643332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698986.643336:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698986.643338:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698986.643342:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6888 00000100:00080000:2.0:1787698986.643346:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698986.643350:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698986.643355:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698986.643367:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134691200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698986.643377:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698986.643382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787698986.643384:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698986.643387:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698986.643392:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6888 00000100:00080000:2.0:1787698986.643397:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698986.643401:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698986.643406:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698986.643420:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134691328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698986.643421:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698986/real 1787698986] req@ffff9d85babd6a00 x1874531134690816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698986.643449:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698986.643453:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134690816/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698986.643455:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698986.643458:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698986.643463:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6888 00000100:00080000:2.0:1787698986.643470:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698986.643473:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698986.643477:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787698986.643479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698986.643485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134690816:192.168.201.137@tcp:250:kworker.0 00000008:00100000:2.0:1787698986.643486:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698986.643491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787698986.643501:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134691456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698986.643509:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698986.643519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698986/real 1787698986] req@ffff9d85babd6680 x1874531134690944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698986.643521:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698986.643529:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698986.643532:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698986.643540:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6888 00000100:00100000:0.0:1787698986.643545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134690944/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698986.643546:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698986.643555:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787698986.643562:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698986.643572:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698986.643576:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787698986.643579:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134691584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698986.643581:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134690944:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698986.643608:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698986.643630:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698986.643648:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698986/real 1787698986] req@ffff9d85babd4380 x1874531134691072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698986.643660:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134691072/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698986.643667:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698986.643670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698986.643673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691072:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698986.643689:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698986.643713:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698986.643724:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698986/real 1787698986] req@ffff9d85babd6300 x1874531134691200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698986.643735:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134691200/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698986.643741:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698986.643744:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698986.643747:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698986.643753:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134691328:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698986.643772:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698986.643781:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698986/real 1787698986] req@ffff9d85babd7480 x1874531134691328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698986.643788:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134691328/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698986.643792:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698986.643794:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698986.643796:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134691328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698986.643799:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134691456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698986.643818:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698986.643828:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698986/real 1787698986] req@ffff9d85babd5c00 x1874531134691456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698986.643838:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134691456/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698986.643845:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698986.643848:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698986.643873:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134691456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698986.643878:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134691584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698986.643896:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698986.643908:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698986/real 1787698986] req@ffff9d85babd4000 x1874531134691584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698986.643919:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134691584/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698986.643926:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698986.643929:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698986.643931:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134691584:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698987.219044:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698988.243035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698989.267040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698990.291025:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698991.316201:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698991.763009:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698991.763016:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698991.763020:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698991.763028:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6893 00000100:00080000:3.0:1787698991.763037:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698991.763043:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787698991.763089:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134691712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698991.763106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698991.763109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698991.763110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698991.763114:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6893 00000100:00080000:3.0:1787698991.763117:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698991.763120:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698991.763129:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698991.763139:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134691712:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698991.763140:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134691840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698991.763155:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698991.763175:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698991.763177:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698991.763182:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6893 00000100:00080000:3.0:1787698991.763187:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698991.763191:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698991.763196:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698991.763204:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787698991.763210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134691968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698991.763220:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698991.763223:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698991.763225:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698991.763228:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6893 00000100:00080000:3.0:1787698991.763232:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698991.763236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698991.763241:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698991.763247:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698991/real 1787698991] req@ffff9d85c0fb3b80 x1874531134691712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698991.763254:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134692096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698991.763270:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698991.763275:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134691712/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698991.763275:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698991.763279:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698991.763287:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6893 00000100:00080000:3.0:1787698991.763292:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698991.763294:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698991.763296:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698991.763301:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698991.763302:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698991.763309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134691712:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787698991.763313:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134692224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698991.763321:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698991.763323:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698991.763324:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698991.763326:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6893 00000100:00080000:3.0:1787698991.763328:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698991.763329:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698991.763332:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698991.763336:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787698991.763339:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134692352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698991.763357:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698991.763360:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698991.763363:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787698991.763364:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698991.763371:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6893 00000100:00080000:3.0:1787698991.763376:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698991.763380:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698991/real 1787698991] req@ffff9d85c0fb0a80 x1874531134691840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698991.763382:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787698991.763390:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698991.763403:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134691840/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698991.763408:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134692480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698991.763438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698991.763442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698991.763447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691840:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698991.763453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698991.763469:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698991.763478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698991/real 1787698991] req@ffff9d85c0fb1c00 x1874531134691968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698991.763488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134691968/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698991.763495:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698991.763499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698991.763502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134691968:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698991.763506:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698991.763520:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698991.763527:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698991/real 1787698991] req@ffff9d85c0fb1f80 x1874531134692096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698991.763538:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134692096/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698991.763545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698991.763548:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698991.763552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698991.763557:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134692224:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698991.763570:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698991.763578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698991/real 1787698991] req@ffff9d85c0fb3100 x1874531134692224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698991.763588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134692224/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698991.763595:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698991.763598:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698991.763600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134692224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698991.763614:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134692352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698991.763630:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698991.763640:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698991/real 1787698991] req@ffff9d85c0fb1880 x1874531134692352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698991.763650:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134692352/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698991.763658:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698991.763661:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698991.763665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134692352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698991.763670:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134692480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698991.763686:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698991.763695:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698991/real 1787698991] req@ffff9d85c0fb1500 x1874531134692480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698991.763706:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134692480/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698991.763714:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698991.763716:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698991.763719:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134692480:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787698992.339027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698993.363532:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698994.386962:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698995.411031:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698996.435039:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698996.883173:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698996.883181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698996.883185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698996.883192:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6898 00000100:00080000:1.0:1787698996.883201:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698996.883206:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787698996.883240:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134692608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698996.883272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698996.883277:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698996.883278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698996.883283:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6898 00000100:00080000:1.0:1787698996.883286:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698996.883288:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698996.883297:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698996.883308:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134692736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698996.883314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134692608:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698996.883317:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698996.883321:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698996.883323:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698996.883326:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6898 00000100:00080000:1.0:1787698996.883328:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698996.883330:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698996.883333:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698996.883340:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134692864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698996.883346:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698996.883348:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698996.883349:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698996.883351:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6898 00000100:00080000:1.0:1787698996.883363:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698996.883365:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698996.883368:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698996.883374:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787698996.883379:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134692992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698996.883390:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698996.883394:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698996.883396:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698996.883400:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6898 00000100:00080000:1.0:1787698996.883404:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698996.883409:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698996.883413:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698996.883416:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698996/real 1787698996] req@ffff9d85b9d2fb80 x1874531134692608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698996.883427:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134693120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698996.883438:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134692608/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698996.883440:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698996.883449:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698996.883453:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698996.883456:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698996.883459:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6898 00000100:00080000:0.0:1787698996.883463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787698996.883466:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698996.883468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134692608:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787698996.883476:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698996.883482:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698996.883490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787698996.883497:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134693248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698996.883510:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698996.883513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787698996.883515:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698996.883517:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698996.883521:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6898 00000100:00100000:0.0:1787698996.883526:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698996/real 1787698996] req@ffff9d85b9d2c000 x1874531134692736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698996.883526:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698996.883530:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787698996.883537:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698996.883542:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134692736/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698996.883548:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134693376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698996.883580:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698996.883584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698996.883589:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692736:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698996.883596:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698996.883616:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698996.883624:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698996/real 1787698996] req@ffff9d85b9d2ce00 x1874531134692864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698996.883631:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134692864/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698996.883636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698996.883638:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698996.883639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692864:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698996.883643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692992:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698996.883653:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698996.883658:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698996/real 1787698996] req@ffff9d85b9d2f800 x1874531134692992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698996.883664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134692992/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698996.883669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698996.883670:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698996.883672:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134692992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698996.883675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134693120:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787698996.883689:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698996.883698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698996/real 1787698996] req@ffff9d85b9d2ca80 x1874531134693120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698996.883709:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134693120/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698996.883716:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698996.883719:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698996.883722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134693120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787698996.883738:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134693248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698996.883755:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698996.883766:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698996/real 1787698996] req@ffff9d85b9d2f480 x1874531134693248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698996.883776:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134693248/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698996.883783:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698996.883786:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698996.883789:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134693248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787698996.883794:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134693376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787698996.883812:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787698996.883822:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698996/real 1787698996] req@ffff9d85b9d2e300 x1874531134693376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698996.883832:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134693376/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698996.883839:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698996.883842:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787698996.883845:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134693376:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787698997.459221:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698998.483017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787698999.507095:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699000.531110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699001.555027:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699002.003007:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699002.003015:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699002.003018:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699002.003029:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6903 00000100:00080000:3.0:1787699002.003039:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699002.003046:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699002.003082:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134693504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699002.003105:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699002.003108:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699002.003109:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699002.003112:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6903 00000100:00080000:3.0:1787699002.003115:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699002.003118:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699002.003125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699002.003134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134693632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699002.003141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699002.003143:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699002.003144:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699002.003146:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6903 00000100:00080000:3.0:1787699002.003148:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699002.003150:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699002.003153:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699002.003161:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134693504:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787699002.003161:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134693760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699002.003178:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699002.003181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699002.003183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699002.003187:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6903 00000100:00080000:3.0:1787699002.003191:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699002.003194:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699002.003199:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699002.003211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134693888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699002.003217:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699002.003219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699002.003221:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699002.003224:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6903 00000100:00080000:3.0:1787699002.003229:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699002.003233:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699002.003238:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699002.003245:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787699002.003249:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134694016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699002.003259:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699002.003262:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699002.003264:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699002.003268:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6903 00000100:00080000:3.0:1787699002.003271:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699002.003274:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699002.003278:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699002.003287:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134694144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699002.003293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699002/real 1787699002] req@ffff9d85c0fb2300 x1874531134693504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699002.003294:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699002.003299:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699002.003302:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699002.003308:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6903 00000100:00080000:3.0:1787699002.003311:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699002.003315:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699002.003318:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699002.003325:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134693504/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699002.003326:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134694272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699002.003353:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699002.003360:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699002.003365:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134693504:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699002.003374:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134693632:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699002.003396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699002.003409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699002/real 1787699002] req@ffff9d85c0fb3480 x1874531134693632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699002.003421:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134693632/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699002.003435:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699002.003438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699002.003442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134693632:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699002.003463:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134693760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699002.003482:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699002.003491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699002/real 1787699002] req@ffff9d85c0fb0700 x1874531134693760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699002.003504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134693760/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699002.003512:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699002.003515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699002.003518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134693760:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699002.003524:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134693888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699002.003544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699002.003555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699002/real 1787699002] req@ffff9d85c0fb1500 x1874531134693888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699002.003566:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134693888/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699002.003574:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699002.003577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699002.003580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134693888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699002.003586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699002.003605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699002.003616:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699002/real 1787699002] req@ffff9d85c0fb1880 x1874531134694016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699002.003628:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134694016/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699002.003636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699002.003639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699002.003642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699002.003648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699002.003665:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699002.003676:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699002/real 1787699002] req@ffff9d85c0fb3100 x1874531134694144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699002.003687:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134694144/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699002.003694:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699002.003697:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699002.003700:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699002.003706:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699002.003724:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699002.003735:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699002/real 1787699002] req@ffff9d85c0fb1f80 x1874531134694272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699002.003746:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134694272/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699002.003753:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699002.003756:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699002.003760:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694272:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699002.579972:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699003.603116:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699004.627024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699005.651057:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699006.675024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699007.123055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699007.123066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699007.123070:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699007.123080:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6908 00000100:00080000:1.0:1787699007.123090:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699007.123097:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699007.123136:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134694400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699007.123167:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699007.123171:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699007.123172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699007.123176:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6908 00000100:00080000:1.0:1787699007.123180:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699007.123184:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699007.123193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699007.123209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134694528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699007.123215:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134694400:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699007.123227:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699007.123231:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699007.123233:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699007.123238:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6908 00000100:00080000:1.0:1787699007.123242:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699007.123245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699007.123251:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699007.123263:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134694656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699007.123272:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699007.123276:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699007.123278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787699007.123282:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699007.123285:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6908 00000100:00080000:1.0:1787699007.123290:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699007.123294:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699007.123299:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699007.123311:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134694784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699007.123319:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699007.123322:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699007.123326:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699007.123327:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699007/real 1787699007] req@ffff9d85b9d2dc00 x1874531134694400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699007.123334:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6908 00000100:00080000:1.0:1787699007.123340:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699007.123347:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699007.123352:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699007.123355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134694400/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699007.123365:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787699007.123366:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134694912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699007.123369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699007.123376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134694400:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699007.123387:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699007.123391:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699007.123393:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699007.123397:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6908 00000100:00080000:1.0:1787699007.123401:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699007.123408:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699007.123409:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134694528:192.168.201.137@tcp:38:kworker.0 00000008:00100000:1.0:1787699007.123414:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699007.123428:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134695040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699007.123438:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699007.123440:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699007.123444:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699007.123447:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699007.123450:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699007/real 1787699007] req@ffff9d85b9d2c380 x1874531134694528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699007.123454:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6908 00000100:00080000:1.0:1787699007.123460:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699007.123463:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699007.123470:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699007.123479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134694528/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699007.123486:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134695168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699007.123504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699007.123509:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699007.123513:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134694528:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699007.123525:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134694656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699007.123548:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699007.123561:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699007/real 1787699007] req@ffff9d85b9d2c700 x1874531134694656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699007.123570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134694656/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699007.123575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699007.123577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699007.123579:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134694656:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699007.123582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134694784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699007.123595:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699007.123600:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699007/real 1787699007] req@ffff9d85b9d2ed80 x1874531134694784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699007.123607:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134694784/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699007.123611:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699007.123612:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699007.123615:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134694784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699007.123618:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694912:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699007.123626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699007.123631:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699007/real 1787699007] req@ffff9d85b9d2d500 x1874531134694912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699007.123640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134694912/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699007.123644:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699007.123646:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699007.123648:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134694912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699007.123661:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699007.123677:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699007.123687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699007/real 1787699007] req@ffff9d85b9d2e680 x1874531134695040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699007.123697:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134695040/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699007.123714:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699007.123718:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699007.123724:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699007.123730:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699007.123748:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699007.123757:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699007/real 1787699007] req@ffff9d85b9d2dc00 x1874531134695168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699007.123764:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134695168/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699007.123769:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699007.123771:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699007.123773:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695168:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699007.699005:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699008.723024:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699009.747031:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699010.771033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699011.794993:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699012.242921:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699012.242932:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699012.242936:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699012.242946:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6914 00000100:00080000:1.0:1787699012.242957:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699012.242976:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699012.243014:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134695296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699012.243053:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699012.243057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699012.243059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699012.243063:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6914 00000100:00080000:1.0:1787699012.243067:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699012.243071:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699012.243080:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699012.243095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134695424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699012.243109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699012.243112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699012.243114:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699012.243117:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6914 00000100:00080000:1.0:1787699012.243121:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699012.243124:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699012.243129:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699012.243151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134695552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699012.243160:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699012.243163:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699012.243165:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699012.243168:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6914 00000100:00080000:1.0:1787699012.243172:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699012.243175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699012.243179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699012.243190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134695680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699012.243198:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699012.243201:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699012.243203:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699012.243207:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6914 00000100:00080000:1.0:1787699012.243211:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699012.243216:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699012.243220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699012.243228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134695808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699012.243236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134695296:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699012.243237:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699012.243248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699012.243250:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699012.243253:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6914 00000100:00080000:1.0:1787699012.243255:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699012.243258:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699012.243261:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699012.243268:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134695936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699012.243274:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699012.243277:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699012.243278:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699012.243279:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6914 00000800:00000100:0.0:1787699012.243334:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699012.243374:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699012.243381:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699012.243386:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699012.243400:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134696064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699012.243404:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699012/real 1787699012] req@ffff9d85b9d2e300 x1874531134695296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699012.243430:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134695296/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699012.243441:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699012.243445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699012.243448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134695296:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699012.243456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134695424:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699012.243472:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699012.243481:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699012/real 1787699012] req@ffff9d85b9d2f480 x1874531134695424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699012.243488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134695424/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699012.243497:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699012.243499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699012.243501:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134695424:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699012.243505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134695552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699012.243514:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699012.243520:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699012/real 1787699012] req@ffff9d85b9d2ca80 x1874531134695552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699012.243526:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134695552/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699012.243531:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699012.243532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699012.243534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134695552:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699012.243536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134695680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699012.243545:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699012.243550:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699012/real 1787699012] req@ffff9d85b9d2f800 x1874531134695680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699012.243564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134695680/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699012.243569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699012.243571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699012.243573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134695680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699012.243591:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695808:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699012.243601:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699012.243607:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699012/real 1787699012] req@ffff9d85b9d2ce00 x1874531134695808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699012.243613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134695808/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699012.243617:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699012.243619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699012.243620:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699012.243623:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699012.243631:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699012.243636:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699012/real 1787699012] req@ffff9d85b9d2c000 x1874531134695936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699012.243642:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134695936/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699012.243652:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699012.243654:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699012.243655:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134695936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699012.243659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699012.243668:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699012.243673:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699012/real 1787699012] req@ffff9d85b9d2fb80 x1874531134696064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699012.243679:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134696064/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699012.243684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699012.243685:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699012.243687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696064:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699012.819049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699013.843055:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699014.867034:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699015.890995:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699016.914995:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699017.362944:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699017.362959:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699017.362962:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699017.362971:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6919 00000100:00080000:1.0:1787699017.362983:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699017.362990:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699017.363026:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134696192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699017.363060:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699017.363063:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699017.363065:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699017.363069:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6919 00000100:00080000:1.0:1787699017.363072:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699017.363075:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699017.363084:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699017.363096:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134696320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699017.363103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134696192:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699017.363110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699017.363114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699017.363115:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699017.363119:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6919 00000100:00080000:1.0:1787699017.363122:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699017.363126:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699017.363132:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699017.363142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134696448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699017.363149:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699017.363151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699017.363152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699017.363154:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6919 00000100:00080000:1.0:1787699017.363159:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699017.363161:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699017.363164:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699017.363170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699017.363173:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134696576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699017.363180:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699017.363183:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699017.363184:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699017.363186:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6919 00000100:00080000:1.0:1787699017.363188:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699017.363190:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699017.363193:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699017.363202:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134696704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699017.363209:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699017.363213:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787699017.363217:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699017/real 1787699017] req@ffff9d85b9d2fb80 x1874531134696192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699017.363217:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699017.363225:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6919 00000100:00080000:1.0:1787699017.363232:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699017.363237:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699017.363242:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699017.363245:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134696192/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699017.363253:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134696832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699017.363264:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699017.363268:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699017.363272:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787699017.363273:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699017.363276:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699017.363281:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134696192:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699017.363281:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6919 00000100:00080000:1.0:1787699017.363285:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699017.363290:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699017.363293:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699017.363299:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134696960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699017.363316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134696320:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699017.363340:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699017.363349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699017/real 1787699017] req@ffff9d85b9d2c000 x1874531134696320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.363357:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134696320/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.363365:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.363368:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699017.363370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134696320:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699017.363373:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134696448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699017.363388:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699017.363398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699017/real 1787699017] req@ffff9d85b9d2ce00 x1874531134696448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.363410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134696448/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.363418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.363421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699017.363425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134696448:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699017.363431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134696576:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699017.363446:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699017.363453:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699017/real 1787699017] req@ffff9d85b9d2f800 x1874531134696576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.363459:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134696576/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.363465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.363467:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699017.363479:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134696576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699017.363484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696704:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699017.363502:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699017.363512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699017/real 1787699017] req@ffff9d85b9d2ca80 x1874531134696704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.363522:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134696704/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.363527:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.363529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699017.363531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696704:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699017.363534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699017.363547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699017.363557:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699017/real 1787699017] req@ffff9d85b9d2f480 x1874531134696832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.363563:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134696832/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.363568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.363570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699017.363571:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699017.363574:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699017.363583:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699017.363588:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699017/real 1787699017] req@ffff9d85b9d2e300 x1874531134696960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.363595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134696960/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.363599:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.363600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699017.363602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134696960:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699017.939043:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699018.962984:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699019.987025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699021.011104:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699022.035097:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699022.482987:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699022.483001:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699022.483005:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699022.483015:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6924 00000100:00080000:1.0:1787699022.483028:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699022.483035:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699022.483075:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134697088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699022.483110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699022.483114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699022.483116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699022.483120:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6924 00000100:00080000:1.0:1787699022.483123:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699022.483126:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699022.483136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699022.483146:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134697216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699022.483154:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699022.483158:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699022.483162:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699022.483163:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134697088:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699022.483169:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6924 00000100:00080000:1.0:1787699022.483174:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699022.483178:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699022.483183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699022.483197:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134697344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699022.483206:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699022.483210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699022.483211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699022.483215:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6924 00000100:00080000:1.0:1787699022.483232:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699022.483239:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787699022.483240:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787699022.483248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699022.483262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134697472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699022.483271:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699022.483274:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699022.483276:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699022.483280:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6924 00000100:00080000:1.0:1787699022.483284:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699022.483289:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699022.483293:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699022/real 1787699022] req@ffff9d85b9d2dc00 x1874531134697088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699022.483301:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699022.483309:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134697088/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699022.483315:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134697600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699022.483320:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699022.483328:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787699022.483329:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699022.483333:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134697088:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699022.483336:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699022.483340:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699022.483345:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6924 00000100:00080000:1.0:1787699022.483348:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699022.483352:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699022.483357:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699022.483363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134697216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699022.483369:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134697728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699022.483376:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699022.483379:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699022.483380:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699022.483383:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6924 00000100:00080000:1.0:1787699022.483386:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699022.483390:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787699022.483391:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:1.0:1787699022.483393:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699022.483403:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134697856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699022.483410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699022/real 1787699022] req@ffff9d85b9d2e680 x1874531134697216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699022.483425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134697216/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699022.483440:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699022.483444:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699022.483447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134697216:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699022.483453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134697344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699022.483475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699022.483486:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699022/real 1787699022] req@ffff9d85b9d2d500 x1874531134697344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699022.483497:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134697344/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699022.483505:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699022.483508:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699022.483511:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134697344:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699022.483516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134697472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699022.483530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699022.483537:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699022/real 1787699022] req@ffff9d85b9d2ed80 x1874531134697472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699022.483560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134697472/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699022.483567:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699022.483570:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699022.483574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134697472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699022.483580:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134697600:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699022.483599:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699022.483620:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699022/real 1787699022] req@ffff9d85b9d2c700 x1874531134697600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699022.483632:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134697600/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699022.483640:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699022.483642:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699022.483646:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134697600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699022.483671:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134697728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699022.483691:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699022.483703:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699022/real 1787699022] req@ffff9d85b9d2c380 x1874531134697728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699022.483714:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134697728/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699022.483721:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699022.483724:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699022.483727:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134697728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699022.483733:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134697856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699022.483751:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699022.483761:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699022/real 1787699022] req@ffff9d85b9d2dc00 x1874531134697856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699022.483773:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134697856/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699022.483780:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699022.483783:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699022.483786:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134697856:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699023.059068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699024.083082:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699025.107055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699026.131024:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699027.155010:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699027.602928:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699027.602937:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699027.602941:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699027.602950:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6929 00000100:00080000:1.0:1787699027.602959:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699027.602965:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699027.602999:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134697984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699027.603023:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699027.603025:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699027.603026:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699027.603029:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6929 00000100:00080000:1.0:1787699027.603032:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699027.603034:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699027.603042:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699027.603051:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134698112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699027.603057:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699027.603059:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699027.603061:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699027.603065:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134697984:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699027.603065:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6929 00000100:00080000:1.0:1787699027.603069:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699027.603074:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699027.603078:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699027.603085:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134698240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699027.603090:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699027.603092:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699027.603093:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699027.603095:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6929 00000100:00080000:1.0:1787699027.603097:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699027.603099:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699027.603101:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699027.603107:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134698368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699027.603113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699027.603114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699027.603116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699027.603118:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6929 00000100:00080000:1.0:1787699027.603120:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699027.603122:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699027.603125:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699027.603131:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699027.603132:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134698496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699027.603139:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699027.603142:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699027.603143:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699027.603146:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6929 00000100:00080000:1.0:1787699027.603150:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699027.603152:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699027.603154:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699027.603161:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134698624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699027.603166:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699027.603168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699027.603169:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699027.603173:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6929 00000100:00100000:0.0:1787699027.603174:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699027/real 1787699027] req@ffff9d85b9d2e300 x1874531134697984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699027.603176:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699027.603187:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699027.603191:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699027.603195:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134697984/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699027.603200:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134698752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699027.603218:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699027.603224:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699027.603229:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134697984:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699027.603247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134698112:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699027.603262:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699027.603269:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699027/real 1787699027] req@ffff9d85b9d2f480 x1874531134698112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699027.603276:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134698112/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699027.603285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699027.603287:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699027.603289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134698112:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699027.603292:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134698240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699027.603300:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699027.603305:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699027/real 1787699027] req@ffff9d85b9d2ca80 x1874531134698240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699027.603311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134698240/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699027.603316:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699027.603317:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699027.603319:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134698240:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699027.603321:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134698368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699027.603329:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699027.603334:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699027/real 1787699027] req@ffff9d85b9d2f800 x1874531134698368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699027.603339:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134698368/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699027.603344:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699027.603345:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699027.603347:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134698368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699027.603350:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134698496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699027.603359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699027.603363:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699027/real 1787699027] req@ffff9d85b9d2ce00 x1874531134698496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699027.603369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134698496/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699027.603373:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699027.603375:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699027.603376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134698496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699027.603379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134698624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699027.603387:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699027.603391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699027/real 1787699027] req@ffff9d85b9d2c000 x1874531134698624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699027.603397:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134698624/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699027.603401:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699027.603402:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699027.603404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134698624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699027.603407:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134698752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699027.603414:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699027.603420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699027/real 1787699027] req@ffff9d85b9d2fb80 x1874531134698752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699027.603426:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134698752/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699027.603430:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699027.603431:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699027.603433:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134698752:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699028.179024:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699029.203283:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699030.227028:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699031.250990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699032.275025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699032.722933:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699032.722940:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699032.722943:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699032.722951:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6934 00000100:00080000:3.0:1787699032.722959:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699032.722964:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699032.722992:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134698880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699032.723012:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699032.723014:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699032.723015:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699032.723018:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6934 00000100:00080000:3.0:1787699032.723019:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699032.723021:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699032.723028:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699032.723036:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134699008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699032.723042:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699032.723044:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699032.723045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699032.723048:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6934 00000100:00080000:3.0:1787699032.723050:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699032.723051:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699032.723055:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699032.723061:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134698880:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787699032.723061:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134699136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699032.723074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699032.723076:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699032.723077:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699032.723079:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6934 00000100:00080000:3.0:1787699032.723081:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699032.723083:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699032.723085:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699032.723092:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134699264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699032.723097:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699032.723099:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699032.723100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699032.723103:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6934 00000100:00080000:3.0:1787699032.723105:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699032.723108:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699032.723110:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699032.723116:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134699392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699032.723120:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699032.723123:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787699032.723127:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699032.723128:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699032.723133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6934 00000100:00080000:3.0:1787699032.723135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699032.723138:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699032.723141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699032.723148:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134699520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699032.723153:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699032.723155:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699032.723156:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699032.723158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6934 00000100:00080000:3.0:1787699032.723160:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699032.723162:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699032.723165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699032.723174:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699032/real 1787699032] req@ffff9d85c0fb1c00 x1874531134698880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699032.723174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134699648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699032.723206:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134698880/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699032.723217:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699032.723221:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699032.723225:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134698880:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699032.723232:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699008:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699032.723252:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699032.723263:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699032/real 1787699032] req@ffff9d85c0fb0a80 x1874531134699008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699032.723274:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134699008/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699032.723288:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699032.723292:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699032.723295:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699008:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699032.723316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699032.723333:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699032.723343:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699032/real 1787699032] req@ffff9d85c0fb3b80 x1874531134699136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699032.723355:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134699136/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699032.723362:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699032.723393:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699032.723397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699136:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699032.723403:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699032.723421:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699032.723431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699032/real 1787699032] req@ffff9d85c0fb0380 x1874531134699264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699032.723441:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134699264/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699032.723449:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699032.723452:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699032.723456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699032.723461:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134699392:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699032.723477:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699032.723492:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699032/real 1787699032] req@ffff9d85c0fb2d80 x1874531134699392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699032.723503:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134699392/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699032.723511:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699032.723514:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699032.723517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134699392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699032.723522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134699520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699032.723538:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699032.723547:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699032/real 1787699032] req@ffff9d85c0fb3800 x1874531134699520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699032.723559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134699520/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699032.723566:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699032.723569:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699032.723572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134699520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699032.723578:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134699648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699032.723593:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699032.723603:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699032/real 1787699032] req@ffff9d85c0fb0000 x1874531134699648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699032.723615:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134699648/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699032.723623:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699032.723625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699032.723628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134699648:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699033.298998:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699034.323007:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699035.347036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699036.370978:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699037.395013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699037.842973:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699037.842980:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699037.842983:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699037.842991:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6939 00000100:00080000:1.0:1787699037.843000:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699037.843005:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699037.843040:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134699776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699037.843065:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699037.843069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699037.843070:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699037.843074:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6939 00000100:00080000:1.0:1787699037.843078:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699037.843082:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699037.843090:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699037.843100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134699904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699037.843107:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134699776:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699037.843109:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699037.843115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699037.843116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699037.843119:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6939 00000100:00080000:1.0:1787699037.843121:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699037.843123:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699037.843127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699037.843134:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134700032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699037.843140:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699037.843142:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699037.843143:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699037.843145:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6939 00000100:00080000:1.0:1787699037.843154:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699037.843157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699037.843160:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699037.843162:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699037.843170:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134700160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699037.843177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699037.843179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699037.843180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699037.843182:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6939 00000100:00080000:1.0:1787699037.843184:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699037.843186:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699037.843189:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699037.843196:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134700288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699037.843202:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699037/real 1787699037] req@ffff9d85b9d2fb80 x1874531134699776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699037.843203:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699037.843210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699037.843211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699037.843213:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6939 00000100:00080000:1.0:1787699037.843215:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699037.843217:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699037.843219:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699037.843225:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134700416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699037.843241:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699037.843243:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787699037.843246:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134699776/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699037.843247:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699037.843253:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6939 00000100:00080000:1.0:1787699037.843259:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699037.843264:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699037.843265:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787699037.843270:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:1.0:1787699037.843273:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699037.843274:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134699776:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699037.843303:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134700544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699037.843311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699904:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699037.843332:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699037.843342:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699037/real 1787699037] req@ffff9d85b9d2c000 x1874531134699904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699037.843354:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134699904/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699037.843367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699037.843370:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699037.843374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134699904:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699037.843380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700032:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699037.843399:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699037.843410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699037/real 1787699037] req@ffff9d85b9d2ce00 x1874531134700032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699037.843418:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134700032/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699037.843425:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699037.843428:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699037.843431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700032:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699037.843436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699037.843454:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699037.843464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699037/real 1787699037] req@ffff9d85b9d2f800 x1874531134700160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699037.843474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134700160/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699037.843479:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699037.843481:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699037.843483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699037.843487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134700288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699037.843500:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699037.843505:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699037/real 1787699037] req@ffff9d85b9d2ca80 x1874531134700288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699037.843512:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134700288/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699037.843516:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699037.843517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699037.843519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134700288:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699037.843522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134700416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699037.843530:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699037.843535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699037/real 1787699037] req@ffff9d85b9d2f480 x1874531134700416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699037.843541:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134700416/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699037.843545:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699037.843547:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699037.843548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134700416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699037.843560:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134700544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699037.843569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699037.843574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699037/real 1787699037] req@ffff9d85b9d2e300 x1874531134700544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699037.843580:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134700544/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699037.843585:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699037.843586:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699037.843588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134700544:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699038.419007:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699039.443017:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699040.466969:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699041.490998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699042.515002:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699042.962956:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699042.962963:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699042.962966:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699042.962974:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6944 00000100:00080000:1.0:1787699042.962984:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699042.962990:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699042.963023:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134700672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699042.963045:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699042.963049:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699042.963051:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699042.963054:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6944 00000100:00080000:1.0:1787699042.963056:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699042.963059:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699042.963067:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699042.963078:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134700800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699042.963083:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134700672:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699042.963087:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699042.963090:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699042.963091:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699042.963094:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6944 00000100:00080000:1.0:1787699042.963096:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699042.963097:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699042.963101:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699042.963107:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134700928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699042.963113:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699042.963115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699042.963116:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699042.963118:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6944 00000100:00080000:1.0:1787699042.963127:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699042.963129:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699042.963132:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699042.963138:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134701056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699042.963143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699042.963145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699042.963147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699042.963150:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6944 00000100:00080000:1.0:1787699042.963152:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699042.963155:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699042.963158:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699042.963164:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699042.963167:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134701184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699042.963176:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699042.963179:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699042.963181:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699042.963186:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6944 00000100:00080000:1.0:1787699042.963201:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699042.963207:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699042.963212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699042.963213:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699042.963226:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134701312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699042.963234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699042.963236:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699042.963238:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699042.963240:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6944 00000100:00080000:1.0:1787699042.963243:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699042.963247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701056:192.168.201.137@tcp:8:kworker.0 00000100:00080000:1.0:1787699042.963248:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699042.963254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699042.963265:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134701440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699042.963285:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134701184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699042.963308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134701312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699042.963321:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134701440:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699043.539044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699044.563052:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699045.587021:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787699046.035116:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787699046.035167:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787699046.035182:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699046.035225:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699046.035234:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699046.035241:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699046.035249:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699046.035256:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699046.035263:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787699046.035348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699042/real 1787699046] req@ffff9d85b9d2d500 x1874531134701184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699046.035372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134701184/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699046.035383:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699046.035388:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699046.035392:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134701184:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699046.035400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699042/real 1787699046] req@ffff9d85b9d2e680 x1874531134701312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699046.035411:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134701312/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699046.035418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699046.035421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699046.035424:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134701312:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699046.035431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699042/real 1787699046] req@ffff9d85b9d2e300 x1874531134701440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699046.035442:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134701440/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699046.035450:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699046.035453:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699046.035456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134701440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699046.035461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699042/real 1787699046] req@ffff9d85b9d2c380 x1874531134700800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699046.035471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134700800/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699046.035478:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699046.035482:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699046.035485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700800:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699046.035491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699042/real 1787699046] req@ffff9d85b9d2c700 x1874531134700928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699046.035501:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134700928/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699046.035507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699046.035511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699046.035514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134700928:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699046.035519:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699042/real 1787699046] req@ffff9d85b9d2ed80 x1874531134701056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699046.035529:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134701056/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699046.035537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699046.035540:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699046.035543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699046.035549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699042/real 1787699046] req@ffff9d85b9d2dc00 x1874531134700672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699046.035560:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134700672/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699046.035568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699046.035571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699046.035574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134700672:192.168.201.137@tcp:250:kworker.0 00010000:00010000:2.0:1787699046.611005:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699047.635023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699048.082952:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699048.082960:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699048.082962:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699048.082971:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6949 00000100:00080000:3.0:1787699048.082979:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699048.082984:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699048.083008:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134701568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699048.083035:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699048.083038:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699048.083040:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699048.083044:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6949 00000100:00080000:3.0:1787699048.083047:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699048.083050:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699048.083059:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699048.083071:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134701696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699048.083082:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134701568:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699048.083082:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699048.083090:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699048.083093:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699048.083097:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6949 00000100:00080000:3.0:1787699048.083100:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699048.083114:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699048.083119:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699048.083131:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134701824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699048.083141:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699048.083144:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699048.083146:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699048.083150:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6949 00000100:00080000:3.0:1787699048.083154:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699048.083159:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699048.083160:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699048.083168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699048.083182:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134701952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699048.083191:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699048.083194:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699048.083195:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699048.083199:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6949 00000100:00100000:0.0:1787699048.083204:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699048/real 1787699048] req@ffff9d85c0fb2a00 x1874531134701568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699048.083205:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699048.083212:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699048.083220:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699048.083234:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134701568/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699048.083235:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134702080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699048.083258:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699048.083259:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699048.083265:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699048.083268:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787699048.083269:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699048.083275:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134701568:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699048.083276:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6949 00000100:00080000:3.0:1787699048.083283:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699048.083289:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699048.083295:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699048.083308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787699048.083308:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134702208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699048.083323:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699048.083327:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699048.083329:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699048.083333:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6949 00000100:00080000:3.0:1787699048.083337:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699048.083342:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699048.083344:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699048.083351:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699048.083359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699048/real 1787699048] req@ffff9d85c0fb2680 x1874531134701696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699048.083367:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134702336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699048.083380:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134701696/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699048.083390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699048.083394:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699048.083398:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701696:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699048.083404:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701824:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699048.083427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699048.083438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699048/real 1787699048] req@ffff9d85c0fb0e00 x1874531134701824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699048.083450:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134701824/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699048.083457:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699048.083461:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699048.083464:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701824:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699048.083472:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699048.083489:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699048.083500:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699048/real 1787699048] req@ffff9d85c0fb0000 x1874531134701952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699048.083510:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134701952/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699048.083517:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699048.083520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699048.083523:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134701952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699048.083528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702080:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699048.083545:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699048.083556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699048/real 1787699048] req@ffff9d85c0fb3800 x1874531134702080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699048.083570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134702080/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699048.083578:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699048.083581:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699048.083584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702080:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699048.083600:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699048.083617:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699048.083627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699048/real 1787699048] req@ffff9d85c0fb2d80 x1874531134702208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699048.083636:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134702208/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699048.083643:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699048.083646:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699048.083649:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702208:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699048.083654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699048.083670:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699048.083678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699048/real 1787699048] req@ffff9d85c0fb0380 x1874531134702336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699048.083688:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134702336/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699048.083705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699048.083709:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699048.083712:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702336:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699048.658990:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699049.683014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699050.706977:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699051.731014:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699052.755022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699053.202949:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699053.202957:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699053.202961:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699053.202979:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6954 00000100:00080000:3.0:1787699053.202994:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699053.203000:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699053.203056:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134702464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699053.203076:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699053.203082:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699053.203083:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699053.203086:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6954 00000100:00080000:3.0:1787699053.203089:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699053.203092:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699053.203099:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699053.203109:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134702592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699053.203113:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134702464:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699053.203121:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699053.203124:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699053.203125:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699053.203127:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6954 00000100:00080000:3.0:1787699053.203129:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699053.203131:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699053.203134:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699053.203141:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134702720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699053.203148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699053.203150:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699053.203151:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699053.203153:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6954 00000100:00080000:3.0:1787699053.203155:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699053.203157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699053.203159:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699053.203166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134702848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699053.203171:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699053.203173:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699053.203176:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787699053.203177:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699053.203180:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6954 00000100:00080000:3.0:1787699053.203184:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699053.203186:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699053.203189:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699053.203195:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134702976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699053.203201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699053.203203:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699053.203204:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699053.203206:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6954 00000100:00080000:3.0:1787699053.203209:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699053.203212:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699053.203215:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699053.203224:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699053/real 1787699053] req@ffff9d85c0fb3b80 x1874531134702464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699053.203224:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134703104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699053.203244:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699053.203248:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699053.203249:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699053.203253:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6954 00000100:00080000:3.0:1787699053.203255:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699053.203259:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699053.203261:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134702464/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699053.203262:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699053.203271:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134703232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699053.203273:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699053.203282:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699053.203289:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134702464:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699053.203313:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134702592:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699053.203336:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699053.203349:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699053/real 1787699053] req@ffff9d85c0fb0a80 x1874531134702592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699053.203361:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134702592/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699053.203370:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699053.203373:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699053.203375:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134702592:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699053.203379:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134702720:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699053.203394:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699053.203400:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699053/real 1787699053] req@ffff9d85c0fb1c00 x1874531134702720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699053.203407:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134702720/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699053.203412:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699053.203413:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699053.203415:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134702720:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699053.203418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134702848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699053.203427:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699053.203431:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699053/real 1787699053] req@ffff9d85c0fb1f80 x1874531134702848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699053.203437:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134702848/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699053.203442:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699053.203443:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699053.203445:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134702848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699053.203448:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702976:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699053.203456:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699053.203461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699053/real 1787699053] req@ffff9d85c0fb3100 x1874531134702976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699053.203467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134702976/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699053.203474:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699053.203477:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699053.203480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134702976:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699053.203485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134703104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699053.203501:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699053.203511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699053/real 1787699053] req@ffff9d85c0fb1880 x1874531134703104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699053.203523:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134703104/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699053.203530:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699053.203533:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699053.203536:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134703104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699053.203542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134703232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699053.203563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699053.203573:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699053/real 1787699053] req@ffff9d85c0fb1500 x1874531134703232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699053.203584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134703232/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699053.203597:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699053.203600:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699053.203603:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134703232:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699053.779038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699054.803018:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699055.827002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699056.851045:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699057.875016:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699058.322996:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699058.323006:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699058.323010:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699058.323020:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6960 00000100:00080000:3.0:1787699058.323030:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699058.323036:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699058.323074:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134703360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699058.323102:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699058.323106:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699058.323108:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699058.323112:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6960 00000100:00080000:3.0:1787699058.323115:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699058.323119:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699058.323127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699058.323137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134703488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699058.323144:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699058.323146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699058.323147:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699058.323152:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6960 00000100:00100000:0.0:1787699058.323155:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134703360:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699058.323155:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699058.323159:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699058.323166:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699058.323174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134703616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699058.323179:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699058.323181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699058.323183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699058.323184:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6960 00000100:00080000:3.0:1787699058.323186:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699058.323188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699058.323192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699058.323205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134703744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699058.323213:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699058.323216:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699058.323218:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787699058.323223:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699058.323226:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6960 00000100:00080000:3.0:1787699058.323231:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699058.323236:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699058.323241:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699058.323253:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134703872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699058.323262:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699058.323267:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699058.323268:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699058.323273:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6960 00000100:00100000:0.0:1787699058.323275:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699058/real 1787699058] req@ffff9d85c0fb0700 x1874531134703360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699058.323276:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699058.323284:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699058.323293:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699058.323302:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134703360/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699058.323307:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134704000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699058.323328:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699058.323329:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699058.323336:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699058.323337:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787699058.323339:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699058.323344:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134703360:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699058.323347:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6960 00000100:00080000:3.0:1787699058.323353:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699058.323358:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699058.323362:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699058.323371:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134703488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787699058.323376:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134704128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699058.323405:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699058.323419:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699058/real 1787699058] req@ffff9d85c0fb3480 x1874531134703488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699058.323430:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134703488/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699058.323438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699058.323441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699058.323444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134703488:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699058.323450:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134703616:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699058.323475:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699058.323485:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699058/real 1787699058] req@ffff9d85c0fb2300 x1874531134703616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699058.323496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134703616/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699058.323504:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699058.323507:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699058.323510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134703616:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699058.323516:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134703744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699058.323533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699058.323543:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699058/real 1787699058] req@ffff9d85c0fb1500 x1874531134703744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699058.323553:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134703744/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699058.323560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699058.323563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699058.323566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134703744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699058.323572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134703872:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699058.323589:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699058.323598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699058/real 1787699058] req@ffff9d85c0fb1880 x1874531134703872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699058.323608:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134703872/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699058.323616:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699058.323619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699058.323622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134703872:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699058.323627:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704000:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699058.323654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699058.323665:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699058/real 1787699058] req@ffff9d85c0fb3100 x1874531134704000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699058.323675:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134704000/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699058.323683:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699058.323685:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699058.323688:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699058.323704:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699058.323719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699058.323729:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699058/real 1787699058] req@ffff9d85c0fb1f80 x1874531134704128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699058.323739:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134704128/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699058.323752:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699058.323755:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699058.323758:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704128:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699058.899099:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699059.922992:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699060.947025:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699061.971086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699062.995094:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699063.442967:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699063.442977:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699063.442981:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699063.442991:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6965 00000100:00080000:1.0:1787699063.443002:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699063.443009:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699063.443054:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134704256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699063.443076:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699063.443079:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699063.443080:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699063.443084:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6965 00000100:00080000:1.0:1787699063.443087:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699063.443089:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699063.443097:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699063.443106:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134704384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699063.443113:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699063.443115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699063.443117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699063.443121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6965 00000100:00100000:0.0:1787699063.443122:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134704256:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699063.443123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699063.443129:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699063.443134:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699063.443142:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134704512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699063.443148:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699063.443151:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699063.443152:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699063.443155:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6965 00000100:00080000:1.0:1787699063.443156:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699063.443158:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699063.443161:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699063.443169:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134704640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699063.443177:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699063.443181:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699063.443183:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699063.443186:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6965 00000100:00080000:1.0:1787699063.443189:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699063.443192:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699063.443197:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699063.443203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699063.443215:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134704768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699063.443224:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699063.443228:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699063.443230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699063.443233:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6965 00000100:00080000:1.0:1787699063.443237:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699063.443241:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699063/real 1787699063] req@ffff9d85b9d2f480 x1874531134704256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699063.443242:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699063.443248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699063.443261:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134704896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699063.443264:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134704256/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699063.443276:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699063.443283:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699063.443285:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699063.443288:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6965 00000100:00080000:0.0:1787699063.443289:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699063.443291:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699063.443296:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699063.443300:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699063.443304:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699063.443310:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134704256:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699063.443314:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134705024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699063.443340:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134704384:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699063.443360:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699063.443368:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699063/real 1787699063] req@ffff9d85b9d2ca80 x1874531134704384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699063.443376:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134704384/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699063.443382:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699063.443384:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699063.443387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134704384:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699063.443393:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134704512:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699063.443411:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699063.443422:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699063/real 1787699063] req@ffff9d85b9d2f800 x1874531134704512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699063.443434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134704512/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699063.443442:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699063.443445:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699063.443448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134704512:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699063.443454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134704640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699063.443471:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699063.443478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699063/real 1787699063] req@ffff9d85b9d2ce00 x1874531134704640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699063.443485:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134704640/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699063.443490:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699063.443492:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699063.443494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134704640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699063.443497:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704768:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699063.443516:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699063.443522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699063/real 1787699063] req@ffff9d85b9d2c000 x1874531134704768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699063.443530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134704768/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699063.443534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699063.443536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699063.443537:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704768:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699063.443541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699063.443551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699063.443555:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699063/real 1787699063] req@ffff9d85b9d2fb80 x1874531134704896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699063.443567:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134704896/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699063.443575:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699063.443577:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699063.443578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134704896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699063.443582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699063.443592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699063.443597:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699063/real 1787699063] req@ffff9d85b9d2dc00 x1874531134705024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699063.443604:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134705024/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699063.443613:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699063.443615:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699063.443616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705024:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699064.019089:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699065.042982:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699066.067067:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699067.091042:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699068.114941:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699068.562964:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699068.562972:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699068.562975:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699068.562985:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6970 00000100:00080000:3.0:1787699068.562995:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699068.563003:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699068.563039:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134705152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699068.563074:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699068.563077:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699068.563078:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699068.563082:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6970 00000100:00080000:3.0:1787699068.563086:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699068.563089:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699068.563097:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699068.563103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134705152:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787699068.563112:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134705280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699068.563123:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699068.563126:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699068.563129:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699068.563133:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6970 00000100:00080000:3.0:1787699068.563137:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699068.563141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699068.563146:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699068.563155:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787699068.563160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134705408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699068.563170:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699068.563174:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699068.563176:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699068.563180:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6970 00000100:00080000:3.0:1787699068.563184:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699068.563188:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699068.563192:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699068.563203:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134705536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699068.563210:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699068/real 1787699068] req@ffff9d85c0fb1c00 x1874531134705152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699068.563215:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699068.563222:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699068.563226:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699068.563233:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6970 00000100:00100000:0.0:1787699068.563237:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134705152/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699068.563238:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699068.563247:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787699068.563248:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699068.563254:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787699068.563254:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699068.563257:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134705152:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787699068.563268:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134705664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699068.563275:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699068.563277:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787699068.563279:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134705280:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699068.563280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699068.563287:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6970 00000100:00080000:3.0:1787699068.563290:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699068.563294:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699068.563299:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699068.563303:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699068.563313:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699068/real 1787699068] req@ffff9d85c0fb0a80 x1874531134705280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699068.563319:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134705792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699068.563322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134705280/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699068.563345:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699068.563347:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699068.563348:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787699068.563354:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787699068.563356:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134705280:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699068.563357:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699068.563366:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134705408:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787699068.563366:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6970 00000100:00080000:3.0:1787699068.563370:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699068.563375:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699068.563380:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699068.563383:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787699068.563393:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134705920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699068.563394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699068/real 1787699068] req@ffff9d85c0fb3b80 x1874531134705408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699068.563425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134705408/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699068.563433:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699068.563437:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699068.563441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134705408:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699068.563447:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134705536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699068.563464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699068.563474:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699068/real 1787699068] req@ffff9d85c0fb0380 x1874531134705536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699068.563482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134705536/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699068.563486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699068.563488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699068.563490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134705536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699068.563494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705664:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699068.563506:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699068.563511:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699068/real 1787699068] req@ffff9d85c0fb2d80 x1874531134705664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699068.563518:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134705664/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699068.563522:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699068.563524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699068.563525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705664:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699068.563536:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699068.563544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699068.563549:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699068/real 1787699068] req@ffff9d85c0fb3800 x1874531134705792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699068.563556:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134705792/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699068.563560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699068.563562:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699068.563563:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699068.563566:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699068.563574:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699068.563578:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699068/real 1787699068] req@ffff9d85c0fb0000 x1874531134705920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699068.563584:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134705920/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699068.563593:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699068.563594:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699068.563596:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134705920:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699069.139015:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699070.162962:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699071.187022:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699072.210969:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699073.235000:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699073.682964:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699073.682971:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699073.682975:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699073.682983:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6975 00000100:00080000:2.0:1787699073.682991:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699073.682997:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699073.683024:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134706048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699073.683045:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699073.683048:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699073.683049:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699073.683051:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6975 00000100:00080000:2.0:1787699073.683053:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699073.683055:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699073.683061:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699073.683068:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134706176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699073.683074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699073.683076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699073.683078:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699073.683080:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6975 00000100:00080000:2.0:1787699073.683082:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699073.683084:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699073.683088:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699073.683095:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134706048:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787699073.683099:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134706304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699073.683110:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699073.683112:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699073.683114:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699073.683118:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6975 00000100:00080000:2.0:1787699073.683121:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699073.683125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699073.683130:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699073.683142:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134706432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699073.683152:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699073.683154:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787699073.683157:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699073.683160:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699073.683166:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6975 00000100:00080000:2.0:1787699073.683169:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699073.683172:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699073.683176:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699073.683183:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134706560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699073.683190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699073.683194:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787699073.683195:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699073/real 1787699073] req@ffff9d85babd7800 x1874531134706048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699073.683195:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699073.683201:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6975 00000100:00080000:2.0:1787699073.683204:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699073.683209:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699073.683214:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699073.683226:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134706048/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699073.683228:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134706688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699073.683246:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699073.683247:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699073.683253:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699073.683254:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787699073.683255:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699073.683260:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134706048:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787699073.683261:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6975 00000100:00080000:2.0:1787699073.683267:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699073.683270:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699073.683272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134706176:192.168.201.137@tcp:38:kworker.0 00000008:00100000:2.0:1787699073.683274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699073.683285:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134706816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699073.683297:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699073.683306:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699073/real 1787699073] req@ffff9d85babd6d80 x1874531134706176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699073.683314:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134706176/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699073.683319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699073.683321:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699073.683323:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134706176:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699073.683339:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134706304:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699073.683350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699073.683355:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699073/real 1787699073] req@ffff9d85babd5500 x1874531134706304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699073.683362:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134706304/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699073.683367:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699073.683368:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699073.683370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134706304:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699073.683373:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134706432:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699073.683381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699073.683394:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699073/real 1787699073] req@ffff9d85babd4000 x1874531134706432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699073.683401:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134706432/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699073.683406:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699073.683408:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699073.683409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134706432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699073.683412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134706560:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699073.683422:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699073.683427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699073/real 1787699073] req@ffff9d85babd5c00 x1874531134706560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699073.683434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134706560/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699073.683438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699073.683440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699073.683441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134706560:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699073.683444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134706688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699073.683452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699073.683457:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699073/real 1787699073] req@ffff9d85babd7480 x1874531134706688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699073.683463:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134706688/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699073.683467:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699073.683469:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699073.683470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134706688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699073.683473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134706816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699073.683481:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699073.683485:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699073/real 1787699073] req@ffff9d85babd6300 x1874531134706816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699073.683491:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134706816/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699073.683499:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699073.683501:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699073.683503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134706816:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699074.259007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699075.283006:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699076.307019:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699077.330963:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699078.355056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699078.802957:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699078.802966:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699078.802970:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699078.802980:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6980 00000100:00080000:2.0:1787699078.802991:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699078.802998:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699078.803031:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134706944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699078.803058:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699078.803066:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699078.803068:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699078.803073:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6980 00000100:00080000:2.0:1787699078.803077:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699078.803080:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699078.803090:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699078.803102:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134707072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699078.803106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134706944:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787699078.803112:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699078.803115:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699078.803116:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699078.803119:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6980 00000100:00080000:2.0:1787699078.803122:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699078.803125:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699078.803129:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699078.803137:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134707200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699078.803143:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699078.803146:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699078.803148:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699078.803150:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6980 00000100:00080000:2.0:1787699078.803152:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699078.803155:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787699078.803159:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:2.0:1787699078.803159:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699078.803166:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134707328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699078.803173:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699078.803176:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699078.803177:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699078.803180:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6980 00000100:00080000:2.0:1787699078.803183:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699078.803186:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699078.803189:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699078.803195:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699078/real 1787699078] req@ffff9d85babd4380 x1874531134706944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699078.803196:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134707456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699078.803205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699078.803207:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699078.803209:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699078.803211:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134706944/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699078.803212:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6980 00000100:00080000:2.0:1787699078.803215:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699078.803218:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787699078.803220:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787699078.803221:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699078.803223:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699078.803227:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134706944:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787699078.803229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134707584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699078.803235:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699078.803237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699078.803239:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699078.803241:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6980 00000100:00080000:2.0:1787699078.803243:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699078.803245:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707072:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787699078.803247:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699078.803251:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699078.803259:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134707712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699078.803264:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699078.803274:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699078/real 1787699078] req@ffff9d85babd6680 x1874531134707072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699078.803284:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134707072/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699078.803291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699078.803294:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699078.803296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707072:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699078.803302:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707200:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699078.803320:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699078.803328:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699078/real 1787699078] req@ffff9d85babd6a00 x1874531134707200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699078.803335:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134707200/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699078.803340:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699078.803342:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699078.803344:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707200:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699078.803347:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699078.803357:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699078.803362:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699078/real 1787699078] req@ffff9d85babd4e00 x1874531134707328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699078.803368:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134707328/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699078.803373:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699078.803374:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699078.803376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699078.803378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134707456:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699078.803386:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699078.803391:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699078/real 1787699078] req@ffff9d85babd7b80 x1874531134707456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699078.803398:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134707456/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699078.803403:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699078.803404:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699078.803406:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134707456:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699078.803408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134707584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699078.803416:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699078.803421:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699078/real 1787699078] req@ffff9d85babd5f80 x1874531134707584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699078.803434:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134707584/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699078.803438:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699078.803440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699078.803442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134707584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699078.803453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134707712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699078.803463:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699078.803468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699078/real 1787699078] req@ffff9d85babd4700 x1874531134707712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699078.803474:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134707712/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699078.803483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699078.803485:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699078.803487:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134707712:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699079.379020:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699080.403022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699081.426996:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699082.451014:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699083.475032:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699083.922950:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699083.922957:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699083.922960:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699083.922967:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6985 00000100:00080000:2.0:1787699083.922984:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699083.922990:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699083.923019:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134707840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699083.923041:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699083.923045:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699083.923047:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699083.923051:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6985 00000100:00080000:2.0:1787699083.923054:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699083.923058:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699083.923067:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699083.923081:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134707968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699083.923092:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699083.923097:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134707840:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787699083.923098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699083.923101:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699083.923109:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6985 00000100:00080000:2.0:1787699083.923113:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699083.923116:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699083.923122:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699083.923132:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134708096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699083.923142:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699083.923145:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699083.923147:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699083.923151:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6985 00000100:00080000:2.0:1787699083.923154:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699083.923158:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699083.923162:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699083.923172:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787699083.923175:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134708224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699083.923186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699083.923193:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699083.923195:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699083.923199:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6985 00000100:00080000:2.0:1787699083.923204:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699083.923208:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699083.923213:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699083.923214:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699083/real 1787699083] req@ffff9d85babd7100 x1874531134707840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699083.923232:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134707840/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699083.923236:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134708352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699083.923247:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699083.923253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699083.923256:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:2.0:1787699083.923260:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699083.923264:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699083.923265:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134707840:192.168.201.137@tcp:250:kworker.0 00000100:00080000:2.0:1787699083.923272:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6985 00000100:00080000:2.0:1787699083.923277:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699083.923280:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699083.923285:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699083.923295:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:2.0:1787699083.923297:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134708480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699083.923310:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699083.923316:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699083.923317:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699083.923321:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6985 00000100:00080000:2.0:1787699083.923324:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699083.923329:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699083.923334:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699083.923336:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787699083.923351:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134708608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699083.923352:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699083/real 1787699083] req@ffff9d85babd4a80 x1874531134707968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699083.923381:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134707968/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699083.923390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699083.923394:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699083.923397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134707968:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699083.923402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708096:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699083.923417:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699083.923424:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699083/real 1787699083] req@ffff9d85babd5180 x1874531134708096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699083.923432:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134708096/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699083.923437:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699083.923439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699083.923441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708096:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699083.923444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708224:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699083.923453:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699083.923458:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699083/real 1787699083] req@ffff9d85babd4700 x1874531134708224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699083.923464:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134708224/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699083.923469:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699083.923471:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699083.923472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699083.923475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134708352:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699083.923483:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699083.923488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699083/real 1787699083] req@ffff9d85babd5f80 x1874531134708352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699083.923494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134708352/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699083.923515:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699083.923517:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699083.923519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134708352:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699083.923530:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134708480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699083.923544:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699083.923550:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699083/real 1787699083] req@ffff9d85babd7b80 x1874531134708480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699083.923557:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134708480/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699083.923564:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699083.923567:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699083.923570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134708480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699083.923575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134708608:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699083.923592:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699083.923602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699083/real 1787699083] req@ffff9d85babd4e00 x1874531134708608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699083.923613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134708608/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699083.923626:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699083.923629:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699083.923632:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134708608:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699084.499087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699085.523046:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699086.547152:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699087.571004:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699088.595025:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699089.042980:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699089.042990:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699089.042994:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699089.043003:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6990 00000100:00080000:0.0:1787699089.043013:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699089.043020:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699089.043059:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134708736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699089.043092:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699089.043096:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699089.043098:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699089.043102:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6990 00000100:00080000:0.0:1787699089.043106:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699089.043111:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699089.043120:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699089.043128:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134708736:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699089.043140:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134708864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699089.043151:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699089.043155:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699089.043158:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699089.043162:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6990 00000100:00080000:0.0:1787699089.043177:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699089.043182:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787699089.043187:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:0.0:1787699089.043188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699089.043202:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134708992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699089.043211:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699089.043215:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699089.043217:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699089.043221:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6990 00000100:00080000:0.0:1787699089.043225:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699089.043231:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699089.043232:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699089/real 1787699089] req@ffff9d85b9d0b800 x1874531134708736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699089.043238:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699089.043254:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134709120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699089.043258:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134708736/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699089.043274:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699089.043281:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699089.043285:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699089.043289:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699089.043291:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6990 00000100:00080000:1.0:1787699089.043295:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787699089.043297:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787699089.043301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134708736:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787699089.043305:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699089.043311:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699089.043322:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699089.043326:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134709248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699089.043339:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699089.043344:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787699089.043344:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787699089.043349:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699089.043355:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6990 00000100:00100000:1.0:1787699089.043357:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699089/real 1787699089] req@ffff9d85b9d09f80 x1874531134708864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699089.043361:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699089.043370:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699089.043372:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134708864/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699089.043379:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787699089.043381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699089.043385:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699089.043395:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134709376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699089.043397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708864:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699089.043407:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708992:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787699089.043427:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699089.043433:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699089.043435:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699089.043440:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6990 00000100:00080000:0.0:1787699089.043446:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787699089.043448:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787699089.043453:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699089.043458:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699089/real 1787699089] req@ffff9d85b9d0a300 x1874531134708992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699089.043460:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699089.043469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134708992/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699089.043477:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699089.043479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699089.043481:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134708992:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699089.043486:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134709504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699089.043489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134709120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699089.043510:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699089.043522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699089/real 1787699089] req@ffff9d85b9d09180 x1874531134709120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699089.043534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134709120/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699089.043542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699089.043545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699089.043548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134709120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699089.043562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134709248:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699089.043581:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699089.043591:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699089/real 1787699089] req@ffff9d85b9d09500 x1874531134709248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699089.043599:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134709248/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699089.043604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699089.043606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699089.043607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134709248:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699089.043611:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134709376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699089.043621:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699089.043627:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699089/real 1787699089] req@ffff9d85b9d08700 x1874531134709376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699089.043634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134709376/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699089.043638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699089.043639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699089.043641:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134709376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699089.043644:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134709504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699089.043651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699089.043656:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699089/real 1787699089] req@ffff9d85b9d09c00 x1874531134709504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699089.043661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134709504/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699089.043666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699089.043667:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699089.043669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134709504:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699089.619035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699090.643015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699091.666982:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699092.691014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699093.715025:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699094.162957:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699094.162964:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699094.162967:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699094.162973:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 6995 00000100:00080000:3.0:1787699094.162989:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699094.162993:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699094.163022:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134709632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699094.163040:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699094.163043:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699094.163045:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699094.163049:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6995 00000100:00080000:3.0:1787699094.163052:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699094.163056:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699094.163065:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699094.163078:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134709760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699094.163082:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134709632:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699094.163094:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699094.163097:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699094.163100:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699094.163104:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6995 00000100:00080000:3.0:1787699094.163108:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699094.163119:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699094.163126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699094.163138:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134709888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787699094.163142:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699094.163150:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699094.163153:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699094.163154:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699094.163158:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 6995 00000100:00080000:3.0:1787699094.163162:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699094.163164:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699094.163168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699094.163176:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134710016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699094.163181:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699094.163184:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699094.163185:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699094.163188:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6995 00000100:00080000:3.0:1787699094.163190:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699094.163192:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699094.163195:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699094.163197:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699094/real 1787699094] req@ffff9d85c0fb0e00 x1874531134709632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699094.163205:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134710144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699094.163218:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787699094.163223:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134709632/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699094.163232:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699094.163237:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699094.163246:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699094.163246:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6995 00000100:00080000:1.0:1787699094.163254:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787699094.163254:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787699094.163260:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134709632:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699094.163262:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699094.163268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699094.163280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134710272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699094.163290:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699094.163294:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787699094.163295:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134709760:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699094.163296:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699094.163305:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 6995 00000100:00080000:3.0:1787699094.163308:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699094.163320:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:1.0:1787699094.163321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787699094.163338:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699094.163350:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134710400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699094.163359:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699094/real 1787699094] req@ffff9d85c0fb2680 x1874531134709760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699094.163373:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134709760/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699094.163381:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699094.163384:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699094.163387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134709760:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699094.163392:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134709888:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699094.163410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699094.163419:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699094/real 1787699094] req@ffff9d85c0fb2a00 x1874531134709888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699094.163430:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134709888/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699094.163437:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699094.163440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699094.163443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134709888:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699094.163448:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699094.163465:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699094.163473:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699094/real 1787699094] req@ffff9d85c0fb0000 x1874531134710016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699094.163482:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134710016/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699094.163488:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699094.163491:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699094.163493:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699094.163498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134710144:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699094.163513:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699094.163521:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699094/real 1787699094] req@ffff9d85c0fb3800 x1874531134710144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699094.163530:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134710144/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699094.163537:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699094.163539:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699094.163542:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134710144:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699094.163547:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134710272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699094.163560:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699094.163568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699094/real 1787699094] req@ffff9d85c0fb2d80 x1874531134710272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699094.163578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134710272/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699094.163585:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699094.163587:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699094.163590:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134710272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699094.163604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134710400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699094.163618:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699094.163626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699094/real 1787699094] req@ffff9d85c0fb0380 x1874531134710400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699094.163635:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134710400/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699094.163642:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699094.163644:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699094.163647:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134710400:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699094.739065:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699095.763103:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699096.787087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699097.810994:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699098.834978:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699099.282956:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699099.282964:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699099.282968:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699099.282977:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7001 00000100:00080000:1.0:1787699099.282987:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699099.282994:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699099.283028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134710528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699099.283050:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699099.283053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699099.283054:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699099.283057:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7001 00000100:00080000:1.0:1787699099.283060:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699099.283062:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699099.283069:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699099.283077:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134710656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699099.283084:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699099.283087:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699099.283089:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699099.283091:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134710528:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699099.283094:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7001 00000100:00080000:1.0:1787699099.283099:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699099.283102:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699099.283105:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699099.283113:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134710784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699099.283119:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699099.283121:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699099.283123:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699099.283125:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7001 00000100:00080000:1.0:1787699099.283143:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699099.283147:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699099.283151:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699099.283155:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699099.283166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134710912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699099.283176:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699099.283179:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699099.283181:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699099.283185:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7001 00000100:00080000:1.0:1787699099.283190:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699099.283195:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283196:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699099/real 1787699099] req@ffff9d85b9d2ed80 x1874531134710528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699099.283199:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699099.283211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134711040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699099.283217:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699099.283219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699099.283220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699099.283221:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7001 00000100:00080000:1.0:1787699099.283224:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699099.283227:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283231:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134710528/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699099.283231:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699099.283257:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134711168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699099.283284:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699099.283287:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699099.283287:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699099.283291:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699099.283296:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:1.0:1787699099.283298:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7001 00000100:00080000:1.0:1787699099.283304:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699099.283305:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134710528:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699099.283309:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699099.283314:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699099.283329:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134711296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699099.283336:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710656:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699099.283359:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283368:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699099/real 1787699099] req@ffff9d85b9d2c700 x1874531134710656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699099.283377:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134710656/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699099.283383:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699099.283385:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699099.283387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710656:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699099.283391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710784:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699099.283401:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283406:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699099/real 1787699099] req@ffff9d85b9d2c380 x1874531134710784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699099.283413:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134710784/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699099.283422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699099.283424:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699099.283426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710784:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699099.283429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699099.283437:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283442:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699099/real 1787699099] req@ffff9d85b9d2e300 x1874531134710912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699099.283447:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134710912/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699099.283452:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699099.283454:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699099.283455:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134710912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699099.283458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711040:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699099.283466:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283471:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699099/real 1787699099] req@ffff9d85b9d2e680 x1874531134711040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699099.283477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134711040/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699099.283482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699099.283483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699099.283486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711040:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699099.283489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699099.283498:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283503:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699099/real 1787699099] req@ffff9d85b9d2d500 x1874531134711168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699099.283509:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134711168/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699099.283514:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699099.283515:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699099.283517:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699099.283528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699099.283536:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699099.283541:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699099/real 1787699099] req@ffff9d85b9d2dc00 x1874531134711296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699099.283546:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134711296/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699099.283551:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699099.283553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699099.283554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711296:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699099.859039:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699100.883038:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699101.907011:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699102.931040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699103.955092:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699104.402968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699104.402978:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699104.402982:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699104.402992:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7006 00000100:00080000:3.0:1787699104.403003:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699104.403010:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699104.403044:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134711424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699104.403064:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699104.403066:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699104.403067:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699104.403070:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7006 00000100:00080000:3.0:1787699104.403073:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699104.403075:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699104.403083:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699104.403102:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134711424:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787699104.403102:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134711552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699104.403115:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699104.403117:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699104.403118:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699104.403121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7006 00000100:00080000:3.0:1787699104.403123:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699104.403125:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699104.403128:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699104.403135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134711680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699104.403145:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699104.403166:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699104.403168:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699104.403171:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7006 00000100:00080000:3.0:1787699104.403174:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699104.403176:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699104.403179:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699104.403189:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134711808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699104.403195:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699104.403201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699104.403205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699104.403207:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699104.403212:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7006 00000100:00080000:3.0:1787699104.403216:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699104.403220:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699104.403225:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699104.403238:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699104/real 1787699104] req@ffff9d85c0fb3b80 x1874531134711424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699104.403239:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134711936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699104.403270:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699104.403274:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787699104.403277:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134711424/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699104.403277:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699104.403287:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7006 00000100:00080000:0.0:1787699104.403291:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699104.403292:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699104.403299:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787699104.403300:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699104.403305:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134711424:192.168.201.137@tcp:250:kworker.0 00000008:00100000:3.0:1787699104.403305:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699104.403318:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134712064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699104.403332:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699104.403335:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699104.403338:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699104.403339:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134711552:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699104.403344:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7006 00000100:00080000:3.0:1787699104.403347:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699104.403349:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699104.403351:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699104.403360:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134712192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699104.403381:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699104.403397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699104/real 1787699104] req@ffff9d85c0fb0a80 x1874531134711552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699104.403410:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134711552/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699104.403418:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699104.403421:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699104.403425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134711552:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699104.403431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134711680:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699104.403450:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699104.403460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699104/real 1787699104] req@ffff9d85c0fb1c00 x1874531134711680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699104.403471:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134711680/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699104.403484:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699104.403487:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699104.403490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134711680:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699104.403496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134711808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699104.403514:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699104.403524:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699104/real 1787699104] req@ffff9d85c0fb1f80 x1874531134711808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699104.403534:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134711808/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699104.403542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699104.403545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699104.403548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134711808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699104.403554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711936:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699104.403571:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699104.403582:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699104/real 1787699104] req@ffff9d85c0fb3100 x1874531134711936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699104.403593:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134711936/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699104.403601:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699104.403604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699104.403607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134711936:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699104.403613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699104.403631:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699104.403640:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699104/real 1787699104] req@ffff9d85c0fb1880 x1874531134712064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699104.403651:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134712064/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699104.403666:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699104.403669:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699104.403673:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699104.403690:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699104.403709:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699104.403719:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699104/real 1787699104] req@ffff9d85c0fb1500 x1874531134712192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699104.403731:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134712192/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699104.403739:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699104.403742:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699104.403745:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712192:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699104.979036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699106.002999:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699107.027029:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699108.051276:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699109.075100:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699109.522917:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699109.522924:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699109.522927:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699109.522934:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7011 00000100:00080000:3.0:1787699109.522942:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699109.522947:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699109.522975:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134712320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699109.522998:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699109.523002:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699109.523003:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699109.523008:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7011 00000100:00080000:3.0:1787699109.523011:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699109.523015:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699109.523023:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699109.523033:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134712448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699109.523051:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699109.523054:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699109.523055:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699109.523058:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7011 00000100:00080000:3.0:1787699109.523061:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699109.523078:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699109.523082:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699109.523096:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134712320:192.168.201.137@tcp:250:kworker.0 00000100:00100000:3.0:1787699109.523096:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134712576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699109.523111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699109.523114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699109.523116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699109.523121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7011 00000100:00080000:3.0:1787699109.523132:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699109.523136:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699109.523141:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699109.523151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134712704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699109.523158:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699109.523160:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699109.523161:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699109.523164:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7011 00000100:00080000:3.0:1787699109.523167:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699109.523171:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699109.523175:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699109.523186:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134712832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699109.523191:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134712448:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699109.523201:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699109.523205:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699109.523207:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699109.523211:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7011 00000100:00080000:3.0:1787699109.523215:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699109.523219:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699109.523223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699109.523234:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134712960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699109.523242:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699109.523245:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699109.523247:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699109.523251:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7011 00000100:00080000:3.0:1787699109.523255:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699109.523259:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699109.523262:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134712576:192.168.201.137@tcp:8:kworker.0 00000008:00100000:3.0:1787699109.523265:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699109.523280:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134713088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699109.523284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134712704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699109.523312:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699109.523351:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699109.523372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713088:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699110.099050:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699111.122987:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699112.147054:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787699112.595145:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787699112.595195:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787699112.595210:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699112.595243:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699112.595252:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699112.595259:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699112.595266:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699112.595273:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699112.595280:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787699112.595342:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699109/real 1787699112] req@ffff9d85c0fb1f80 x1874531134713088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699112.595373:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134713088/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699112.595397:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699112.595403:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699112.595408:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699112.595417:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699109/real 1787699112] req@ffff9d85c0fb0700 x1874531134712576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699112.595430:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134712576/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699112.595439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699112.595442:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699112.595446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134712576:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699112.595455:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699109/real 1787699112] req@ffff9d85c0fb1500 x1874531134712704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699112.595467:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134712704/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699112.595475:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699112.595479:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699112.595482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134712704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699112.595488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699109/real 1787699112] req@ffff9d85c0fb1880 x1874531134712832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699112.595500:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134712832/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699112.595507:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699112.595511:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699112.595514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712832:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699112.595520:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699109/real 1787699112] req@ffff9d85c0fb3100 x1874531134712960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699112.595532:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134712960/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699112.595539:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699112.595542:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699112.595546:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134712960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699112.595552:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699109/real 1787699112] req@ffff9d85c0fb2300 x1874531134712320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699112.595564:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134712320/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699112.595571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699112.595574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699112.595577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134712320:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699112.595583:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699109/real 1787699112] req@ffff9d85c0fb3480 x1874531134712448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699112.595594:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134712448/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699112.595601:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699112.595604:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699112.595607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134712448:192.168.201.137@tcp:38:kworker.0 00010000:00010000:2.0:1787699113.171007:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699114.195011:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699114.643396:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699114.643407:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699114.643410:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699114.643420:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7016 00000100:00080000:2.0:1787699114.643435:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699114.643443:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699114.643490:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134713216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699114.643521:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699114.643524:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699114.643526:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699114.643531:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7016 00000100:00080000:2.0:1787699114.643534:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699114.643538:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699114.643547:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699114.643559:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134713344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699114.643569:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699114.643572:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699114.643574:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699114.643577:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7016 00000100:00080000:2.0:1787699114.643581:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699114.643586:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699114.643590:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699114.643601:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134713472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699114.643609:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699114.643612:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699114.643613:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699114.643616:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7016 00000100:00080000:2.0:1787699114.643620:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699114.643623:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699114.643628:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699114.643637:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134713600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699114.643646:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699114.643649:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699114.643651:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699114.643654:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7016 00000100:00080000:2.0:1787699114.643657:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699114.643661:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699114.643665:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699114.643675:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134713728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699114.643682:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699114.643686:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699114.643687:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699114.643690:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7016 00000100:00080000:2.0:1787699114.643693:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699114.643696:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699114.643700:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699114.643719:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134713856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699114.643728:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699114.643731:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699114.643732:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699114.643735:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7016 00000100:00080000:2.0:1787699114.643738:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699114.643742:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699114.643746:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699114.643759:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134713984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699114.644592:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134713216:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699114.644658:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699114.644695:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699114/real 1787699114] req@ffff9d85babd6a00 x1874531134713216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699114.644712:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134713216/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699114.644724:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699114.644728:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699114.644732:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134713216:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699114.644739:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134713344:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699114.644757:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699114.644767:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699114/real 1787699114] req@ffff9d85babd6680 x1874531134713344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699114.644777:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134713344/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699114.644785:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699114.644788:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699114.644792:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134713344:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699114.644798:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134713472:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699114.644813:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699114.644821:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699114/real 1787699114] req@ffff9d85babd4380 x1874531134713472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699114.644831:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134713472/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699114.644839:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699114.644842:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699114.644845:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134713472:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699114.644874:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134713600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699114.644892:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699114.644901:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699114/real 1787699114] req@ffff9d85babd6300 x1874531134713600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699114.644912:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134713600/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699114.644920:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699114.644923:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699114.644926:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134713600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699114.644932:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713728:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699114.644946:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699114.644954:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699114/real 1787699114] req@ffff9d85babd7480 x1874531134713728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699114.644964:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134713728/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699114.644972:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699114.644985:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699114.644988:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713728:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699114.644994:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699114.645011:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699114.645020:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699114/real 1787699114] req@ffff9d85babd5c00 x1874531134713856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699114.646194:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134713856/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699114.646213:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699114.646217:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699114.646220:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699114.646229:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699114.646261:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699114.646275:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699114/real 1787699114] req@ffff9d85babd4000 x1874531134713984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699114.646286:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134713984/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699114.646295:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699114.646298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699114.646301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134713984:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699115.219089:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699116.243105:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699117.267352:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699118.290987:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699119.315047:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699119.765044:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699119.765053:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699119.765057:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699119.765067:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7021 00000100:00080000:1.0:1787699119.765079:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699119.765086:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699119.765117:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134714112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699119.765138:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699119.765141:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699119.765142:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699119.765145:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7021 00000100:00080000:1.0:1787699119.765148:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699119.765150:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699119.765158:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699119.765166:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134714240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699119.765172:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699119.765174:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699119.765175:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699119.765178:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7021 00000100:00080000:1.0:1787699119.765180:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699119.765181:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699119.765185:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699119.765188:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134714112:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699119.765194:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134714368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699119.765204:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699119.765206:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699119.765208:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699119.765212:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7021 00000100:00080000:1.0:1787699119.765215:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699119.765218:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699119.765223:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699119.765235:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134714496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699119.765245:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699119.765249:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787699119.765250:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699119.765252:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699119.765259:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7021 00000100:00080000:1.0:1787699119.765264:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699119.765268:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699119.765273:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699119.765286:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699119/real 1787699119] req@ffff9d85b9d2fb80 x1874531134714112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699119.765288:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134714624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699119.765311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134714112/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699119.765317:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699119.765321:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699119.765323:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699119.765328:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7021 00000100:00080000:1.0:1787699119.765333:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699119.765338:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787699119.765343:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787699119.765343:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699119.765352:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699119.765357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134714112:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699119.765366:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134714240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699119.765371:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134714752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699119.765382:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699119.765385:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699119.765387:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787699119.765392:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699119.765394:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7021 00000100:00080000:1.0:1787699119.765400:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699119.765404:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699119.765410:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699119/real 1787699119] req@ffff9d85b9d2c000 x1874531134714240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699119.765410:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699119.765425:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134714240/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699119.765434:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787699119.765439:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134714880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699119.765440:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699119.765448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134714240:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699119.765474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134714368:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699119.765493:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699119.765503:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699119/real 1787699119] req@ffff9d85b9d2ce00 x1874531134714368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699119.765513:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134714368/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699119.765521:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699119.765524:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699119.765527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134714368:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699119.765532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134714496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699119.765546:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699119.765554:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699119/real 1787699119] req@ffff9d85b9d2f800 x1874531134714496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699119.765565:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134714496/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699119.765571:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699119.765574:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699119.765577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134714496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699119.765583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134714624:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699119.765599:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699119.765607:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699119/real 1787699119] req@ffff9d85b9d2ca80 x1874531134714624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699119.765618:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134714624/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699119.765625:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699119.765628:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699119.765631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134714624:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699119.765636:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134714752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699119.765651:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699119.765659:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699119/real 1787699119] req@ffff9d85b9d2f480 x1874531134714752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699119.765670:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134714752/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699119.765678:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699119.765681:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699119.765684:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134714752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699119.765688:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134714880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699119.765703:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699119.765711:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699119/real 1787699119] req@ffff9d85b9d2dc00 x1874531134714880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699119.765721:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134714880/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699119.765729:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699119.765732:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699119.765734:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134714880:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787699120.339035:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699121.363031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699122.387073:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699123.411031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699124.434997:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699124.885388:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699124.885397:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699124.885401:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699124.885411:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7026 00000100:00080000:3.0:1787699124.885421:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699124.885427:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699124.885462:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134715008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699124.885489:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699124.885493:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699124.885495:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699124.885499:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7026 00000100:00080000:3.0:1787699124.885502:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699124.885505:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699124.885514:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699124.885527:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134715136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699124.885537:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699124.885542:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787699124.885543:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134715008:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699124.885545:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699124.885553:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7026 00000100:00080000:3.0:1787699124.885558:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699124.885562:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699124.885567:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699124.885579:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134715264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699124.885588:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699124.885591:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699124.885593:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699124.885597:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7026 00000100:00080000:3.0:1787699124.885601:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699124.885604:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699124.885608:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699124.885613:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787699124.885622:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134715392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699124.885632:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699124.885636:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699124.885637:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699124.885642:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7026 00000100:00080000:3.0:1787699124.885646:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699124.885650:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699124.885655:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699124.885657:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699124/real 1787699124] req@ffff9d85c0fb1c00 x1874531134715008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.885667:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134715520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699124.885679:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134715008/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699124.885679:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699124.885687:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699124.885700:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699124.885705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699124.885707:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7026 00000100:00080000:0.0:1787699124.885712:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699124.885717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134715008:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699124.885719:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699124.885725:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699124.885730:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699124.885740:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134715648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699124.885744:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134715136:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699124.885758:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699124.885761:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699124.885763:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699124.885768:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7026 00000100:00080000:3.0:1787699124.885772:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699124.885777:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699124.885779:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699124.885785:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699124.885795:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699124/real 1787699124] req@ffff9d85c0fb0a80 x1874531134715136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.885799:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134715776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699124.885829:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134715136/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699124.885880:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699124.885884:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699124.885887:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134715136:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699124.885895:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134715264:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699124.885914:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699124.885924:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699124/real 1787699124] req@ffff9d85c0fb3b80 x1874531134715264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699124.885931:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134715264/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699124.885937:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699124.885939:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699124.885941:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134715264:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699124.885945:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134715392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699124.885960:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699124.885969:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699124/real 1787699124] req@ffff9d85c0fb0380 x1874531134715392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699124.885987:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134715392/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699124.885995:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699124.885998:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699124.886001:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134715392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699124.886006:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134715520:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699124.886023:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699124.886030:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699124/real 1787699124] req@ffff9d85c0fb2d80 x1874531134715520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699124.886037:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134715520/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699124.886042:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699124.886044:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699124.886046:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134715520:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699124.886057:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134715648:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699124.886067:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699124.886072:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699124/real 1787699124] req@ffff9d85c0fb3800 x1874531134715648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699124.886079:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134715648/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699124.886083:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699124.886085:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699124.886086:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134715648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699124.886089:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134715776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699124.886098:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699124.886102:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699124/real 1787699124] req@ffff9d85c0fb0000 x1874531134715776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699124.886109:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134715776/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699124.886114:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699124.886115:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699124.886117:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134715776:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699125.459085:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699126.482976:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699127.507016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699128.534282:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699129.558408:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699130.002921:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699130.002932:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699130.002937:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699130.002948:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7031 00000100:00080000:2.0:1787699130.002959:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699130.002968:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699130.003011:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134715904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699130.003044:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699130.003047:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699130.003049:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699130.003054:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7031 00000100:00080000:2.0:1787699130.003057:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699130.003061:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699130.003070:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699130.003208:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134715904:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699130.003268:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699130.003307:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699130/real 1787699130] req@ffff9d85babd5500 x1874531134715904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699130.003324:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134715904/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699130.003341:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699130.003345:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699130.003348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134715904:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787699130.003735:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134716032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699130.003748:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699130.003751:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699130.003753:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699130.003757:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7031 00000100:00080000:2.0:1787699130.003761:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699130.003765:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699130.003771:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699130.003781:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134716160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699130.003790:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699130.003793:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699130.003795:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699130.003798:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7031 00000100:00080000:2.0:1787699130.003800:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699130.003804:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699130.003808:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699130.003818:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134716288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699130.003827:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699130.003830:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699130.003831:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699130.003834:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7031 00000100:00080000:2.0:1787699130.003837:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699130.003841:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699130.003845:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699130.003874:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134716416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699130.003883:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699130.003886:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699130.003887:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699130.003891:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7031 00000100:00080000:2.0:1787699130.003895:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699130.003898:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699130.003902:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699130.003912:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134716544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699130.003920:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699130.003923:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699130.003924:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699130.003928:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7031 00000100:00080000:2.0:1787699130.003931:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699130.003934:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699130.003938:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699130.003948:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134716672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699130.005709:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716032:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699130.005762:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699130.005782:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699130/real 1787699130] req@ffff9d85babd6d80 x1874531134716032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699130.005798:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134716032/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699130.005808:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699130.005812:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699130.005821:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716032:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699130.005828:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716160:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699130.005882:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699130.005894:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699130/real 1787699130] req@ffff9d85babd7800 x1874531134716160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699130.005906:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134716160/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699130.005914:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699130.005918:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699130.005921:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716160:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699130.005928:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716288:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699130.005945:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699130.005955:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699130/real 1787699130] req@ffff9d85babd5500 x1874531134716288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699130.005966:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134716288/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699130.005974:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699130.005977:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699130.005980:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699130.005986:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134716416:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699130.006003:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699130.006013:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699130/real 1787699130] req@ffff9d85babd4000 x1874531134716416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699130.006023:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134716416/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699130.006029:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699130.006032:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699130.006035:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134716416:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699130.006041:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134716544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699130.006059:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699130.006069:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699130/real 1787699130] req@ffff9d85babd5c00 x1874531134716544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699130.006081:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134716544/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699130.006089:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699130.006092:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699130.006095:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134716544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699130.006101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134716672:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699130.006117:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699130.006126:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699130/real 1787699130] req@ffff9d85babd7480 x1874531134716672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699130.006137:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134716672/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699130.006145:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699130.006148:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699130.006151:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134716672:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699130.579029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699131.603101:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699132.626990:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699133.651114:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699134.675013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699135.125503:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699135.125515:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699135.125519:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699135.125530:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7036 00000100:00080000:3.0:1787699135.125541:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699135.125548:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699135.125588:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134716800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699135.125684:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134716800:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699135.125747:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699135.125779:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699135.125783:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699135.125785:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699135.125790:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7036 00000100:00080000:3.0:1787699135.125795:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699135.125799:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699135.125808:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699135.125823:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134716928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699135.125910:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699135/real 1787699135] req@ffff9d85c0fb2a00 x1874531134716800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699135.125929:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134716800/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699135.125944:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699135.125948:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699135.125952:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134716800:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699135.125973:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716928:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699135.125994:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699135.126007:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699135/real 1787699135] req@ffff9d85c0fb2680 x1874531134716928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699135.126018:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134716928/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699135.126027:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699135.126030:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699135.126033:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134716928:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699135.126111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699135.126115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699135.126117:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699135.126121:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7036 00000100:00080000:3.0:1787699135.126126:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699135.126130:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699135.126136:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699135.126147:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134717056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699135.126173:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699135.126176:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699135.126178:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699135.126181:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7036 00000100:00080000:3.0:1787699135.126185:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699135.126189:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699135.126194:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699135.126197:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787699135.126209:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134717184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699135.126219:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699135.126223:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699135.126227:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699135.126230:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699135.126235:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7036 00000100:00100000:0.0:1787699135.126238:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699135/real 1787699135] req@ffff9d85c0fb0e00 x1874531134717056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699135.126241:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699135.126250:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699135.126254:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134717056/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699135.126257:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699135.126264:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699135.126268:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699135.126271:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717056:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787699135.126277:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134717312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699135.126293:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717184:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787699135.126297:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699135.126302:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699135.126304:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699135.126308:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7036 00000100:00080000:3.0:1787699135.126313:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699135.126317:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000800:00000100:0.0:1787699135.126321:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000008:00100000:3.0:1787699135.126324:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699135.126337:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699135/real 1787699135] req@ffff9d85c0fb2680 x1874531134717184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699135.126339:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134717440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699135.126364:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699135.126369:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787699135.126370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134717184/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699135.126371:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699135.126379:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7036 00000100:00080000:3.0:1787699135.126385:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699135.126390:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699135.126391:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:0.0:1787699135.126398:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000008:00100000:3.0:1787699135.126399:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699135.126404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787699135.126416:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134717568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699135.126418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134717312:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699135.126442:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699135.126452:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699135/real 1787699135] req@ffff9d85c0fb2a00 x1874531134717312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699135.126461:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134717312/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699135.126466:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699135.126468:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699135.126470:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134717312:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699135.126474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134717440:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699135.126490:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699135.126498:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699135/real 1787699135] req@ffff9d85c0fb0000 x1874531134717440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699135.126510:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134717440/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699135.126519:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699135.126522:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699135.126525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134717440:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699135.126539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134717568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699135.126557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699135.126568:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699135/real 1787699135] req@ffff9d85c0fb3800 x1874531134717568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699135.126590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134717568/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699135.126602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699135.126605:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699135.126608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134717568:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699135.699070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699136.723024:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699137.747036:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699138.770988:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699139.795070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699140.242957:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699140.242966:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699140.242970:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699140.242978:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7042 00000100:00080000:3.0:1787699140.242989:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699140.242995:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699140.243028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134717696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699140.243054:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699140.243057:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699140.243059:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699140.243063:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7042 00000100:00080000:3.0:1787699140.243070:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699140.243073:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699140.243081:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699140.243091:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134717824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699140.243100:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699140.243103:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699140.243105:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699140.243110:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134717696:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699140.243111:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7042 00000100:00080000:3.0:1787699140.243116:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699140.243121:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699140.243126:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699140.243135:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134717952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699140.243143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699140.243146:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699140.243148:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699140.243151:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7042 00000100:00080000:3.0:1787699140.243154:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699140.243157:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699140.243167:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699140.243173:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787699140.243182:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134718080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699140.243192:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699140.243195:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699140.243197:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699140.243200:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7042 00000100:00080000:3.0:1787699140.243203:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699140.243207:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699140.243211:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699140.243216:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699140/real 1787699140] req@ffff9d85c0fb2d80 x1874531134717696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699140.243221:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134718208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699140.243241:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699140.243244:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699140.243246:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699140.243248:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134717696/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699140.243252:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7042 00000100:00080000:3.0:1787699140.243257:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699140.243263:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699140.243270:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699140.243281:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134718336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699140.243282:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699140.243291:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00080000:3.0:1787699140.243294:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699140.243299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134717696:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699140.243300:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699140.243303:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699140.243309:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7042 00000100:00080000:3.0:1787699140.243312:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699140.243316:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699140.243319:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699140.243327:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717824:192.168.201.137@tcp:38:kworker.0 00000100:00100000:3.0:1787699140.243330:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134718464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699140.243350:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699140.243361:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699140/real 1787699140] req@ffff9d85c0fb0380 x1874531134717824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699140.243370:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134717824/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699140.243378:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699140.243381:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699140.243384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717824:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699140.243389:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717952:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699140.243408:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699140.243418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699140/real 1787699140] req@ffff9d85c0fb3b80 x1874531134717952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699140.243429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134717952/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699140.243437:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699140.243439:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699140.243442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134717952:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699140.243446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699140.243461:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699140.243470:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699140/real 1787699140] req@ffff9d85c0fb0a80 x1874531134718080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699140.243481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134718080/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699140.243489:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699140.243492:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699140.243495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699140.243500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134718208:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699140.243515:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699140.243522:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699140/real 1787699140] req@ffff9d85c0fb1c00 x1874531134718208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699140.243529:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134718208/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699140.243534:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699140.243536:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699140.243538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134718208:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699140.243563:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134718336:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699140.243576:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699140.243583:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699140/real 1787699140] req@ffff9d85c0fb3480 x1874531134718336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699140.243590:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134718336/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699140.243595:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699140.243597:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699140.243598:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134718336:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699140.243609:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134718464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699140.243618:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699140.243623:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699140/real 1787699140] req@ffff9d85c0fb2300 x1874531134718464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699140.243630:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134718464/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699140.243634:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699140.243636:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699140.243637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134718464:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699140.819040:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699141.843004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699142.867063:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699143.891050:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699144.915052:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699145.362968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699145.362979:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699145.362983:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699145.362993:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7047 00000100:00080000:3.0:1787699145.363005:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699145.363012:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699145.363052:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134718592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699145.363077:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699145.363080:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699145.363081:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699145.363084:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7047 00000100:00080000:3.0:1787699145.363087:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699145.363089:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699145.363098:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699145.363107:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134718720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699145.363117:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699145.363120:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699145.363122:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699145.363127:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134718592:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699145.363129:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7047 00000100:00080000:3.0:1787699145.363135:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699145.363141:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699145.363147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699145.363160:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134718848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699145.363168:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699145.363171:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699145.363172:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699145.363174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7047 00000100:00080000:3.0:1787699145.363177:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699145.363179:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699145.363184:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699145.363192:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134718976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699145.363198:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699145.363208:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699145.363210:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699145.363211:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699145.363213:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7047 00000100:00080000:3.0:1787699145.363216:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699145.363219:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699145.363221:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699145.363228:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134719104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699145.363233:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699145.363235:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699145.363237:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699145.363240:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7047 00000100:00080000:3.0:1787699145.363242:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699145.363245:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699145.363248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699145.363251:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699145/real 1787699145] req@ffff9d85c0fb3100 x1874531134718592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699145.363259:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134719232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699145.363265:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699145.363268:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787699145.363269:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134718592/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699145.363270:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699145.363278:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7047 00000100:00080000:3.0:1787699145.363282:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699145.363288:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699145.363293:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699145.363298:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699145.363302:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134719360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699145.363304:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699145.363311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134718592:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699145.363340:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718720:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699145.363361:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699145.363374:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699145/real 1787699145] req@ffff9d85c0fb1880 x1874531134718720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699145.363388:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134718720/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699145.363393:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699145.363395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699145.363397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718720:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699145.363401:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718848:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699145.363414:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699145.363420:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699145/real 1787699145] req@ffff9d85c0fb1500 x1874531134718848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699145.363427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134718848/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699145.363431:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699145.363433:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699145.363435:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718848:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699145.363438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699145.363448:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699145.363468:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699145/real 1787699145] req@ffff9d85c0fb0700 x1874531134718976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699145.363477:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134718976/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699145.363482:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699145.363484:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699145.363485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134718976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699145.363489:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134719104:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699145.363504:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699145.363515:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699145/real 1787699145] req@ffff9d85c0fb1f80 x1874531134719104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699145.363525:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134719104/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699145.363532:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699145.363535:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699145.363538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134719104:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699145.363544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134719232:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699145.363563:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699145.363574:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699145/real 1787699145] req@ffff9d85c0fb2300 x1874531134719232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699145.363585:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134719232/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699145.363591:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699145.363594:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699145.363597:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134719232:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699145.363602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134719360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699145.363619:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699145.363628:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699145/real 1787699145] req@ffff9d85c0fb3480 x1874531134719360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699145.363638:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134719360/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699145.363646:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699145.363649:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699145.363652:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134719360:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699145.939075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699146.963043:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699147.987074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699149.011094:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699150.035064:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699150.482994:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699150.483002:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699150.483006:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699150.483016:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7052 00000100:00080000:0.0:1787699150.483028:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699150.483035:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699150.483075:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134719488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699150.483100:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699150.483102:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699150.483103:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699150.483106:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7052 00000100:00080000:0.0:1787699150.483108:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699150.483110:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699150.483118:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699150.483126:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134719616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699150.483132:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699150.483135:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699150.483136:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699150.483138:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7052 00000100:00080000:0.0:1787699150.483140:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699150.483142:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699150.483145:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699150.483152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134719744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699150.483153:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134719488:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787699150.483165:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699150.483168:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699150.483169:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699150.483171:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7052 00000100:00080000:0.0:1787699150.483173:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699150.483174:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699150.483177:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699150.483185:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134719872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699150.483190:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699150.483192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699150.483193:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699150.483194:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7052 00000100:00080000:0.0:1787699150.483196:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699150.483199:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699150.483201:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699150.483208:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134720000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699150.483213:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699150.483215:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699150.483216:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699150.483218:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7052 00000100:00080000:0.0:1787699150.483223:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699150.483226:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699150.483230:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699150.483236:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699150.483242:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134720128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699150.483258:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699150.483261:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699150.483263:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699150.483267:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7052 00000100:00080000:0.0:1787699150.483271:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699150.483274:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699150.483279:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699150.483288:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699150/real 1787699150] req@ffff9d85b9d0a680 x1874531134719488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699150.483289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134720256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699150.483322:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134719488/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699150.483337:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699150.483341:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699150.483346:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134719488:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699150.483353:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134719616:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699150.483388:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699150.483398:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699150/real 1787699150] req@ffff9d85b9d0b480 x1874531134719616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699150.483406:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134719616/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699150.483411:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699150.483414:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699150.483417:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134719616:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699150.483443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134719744:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699150.483464:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699150.483474:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699150/real 1787699150] req@ffff9d85b9d08e00 x1874531134719744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699150.483481:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134719744/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699150.483486:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699150.483488:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699150.483490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134719744:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699150.483494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134719872:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699150.483505:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699150.483510:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699150/real 1787699150] req@ffff9d85b9d08000 x1874531134719872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699150.483517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134719872/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699150.483521:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699150.483523:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699150.483524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134719872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699150.483527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720000:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699150.483539:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699150.483544:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699150/real 1787699150] req@ffff9d85b9d08a80 x1874531134720000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699150.483550:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134720000/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699150.483555:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699150.483556:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699150.483558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720000:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699150.483560:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720128:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699150.483569:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699150.483573:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699150/real 1787699150] req@ffff9d85b9d0ad80 x1874531134720128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699150.483579:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134720128/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699150.483583:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699150.483585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699150.483587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720128:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699150.483590:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720256:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699150.483598:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699150.483602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699150/real 1787699150] req@ffff9d85b9d0bb80 x1874531134720256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699150.483613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134720256/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699150.483618:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699150.483619:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699150.483620:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720256:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787699151.058955:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699152.084252:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699153.107039:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699154.131143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699155.156633:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699155.603032:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699155.603041:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699155.603044:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699155.603052:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7057 00000100:00080000:1.0:1787699155.603061:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699155.603066:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699155.603098:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134720384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699155.603147:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134720384:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699155.603203:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699155.603305:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699155.603309:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699155.603311:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699155.603324:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7057 00000100:00080000:1.0:1787699155.603329:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699155.603332:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699155.603340:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699155.603350:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134720512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699155.603397:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699155/real 1787699155] req@ffff9d85b9d2d500 x1874531134720384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699155.603409:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134720384/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699155.603416:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699155.603418:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699155.603421:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134720384:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699155.603438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134720512:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699155.603453:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699155.603461:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699155/real 1787699155] req@ffff9d85b9d2e680 x1874531134720512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699155.603469:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134720512/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699155.603474:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699155.603476:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699155.603478:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134720512:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787699155.603586:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699155.603590:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699155.603592:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699155.603596:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7057 00000100:00080000:1.0:1787699155.603599:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699155.603602:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699155.603605:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699155.603614:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134720640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699155.603621:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699155.603623:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699155.603625:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699155.603626:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7057 00000100:00080000:1.0:1787699155.603628:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699155.603630:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699155.603632:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699155.603637:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134720768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699155.603642:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699155.603644:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699155.603645:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699155.603647:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7057 00000100:00080000:1.0:1787699155.603649:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699155.603660:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699155.603662:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699155.603668:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134720896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699155.603673:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699155.603675:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699155.603676:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699155.603678:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7057 00000100:00080000:1.0:1787699155.603679:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699155.603682:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699155.603684:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699155.603689:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134721024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699155.603693:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699155.603695:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699155.603695:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699155.603697:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7057 00000100:00080000:1.0:1787699155.603699:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699155.603701:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699155.603704:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699155.603713:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134721152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699155.603772:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134720640:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699155.603797:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699155.603805:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699155/real 1787699155] req@ffff9d85b9d2e300 x1874531134720640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699155.603813:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134720640/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699155.603818:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699155.603820:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699155.603822:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134720640:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699155.603826:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134720768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699155.603836:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699155.603841:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699155/real 1787699155] req@ffff9d85b9d2c380 x1874531134720768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699155.603847:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134720768/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699155.603879:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699155.603882:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699155.603884:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134720768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699155.603888:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720896:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699155.603904:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699155.603915:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699155/real 1787699155] req@ffff9d85b9d2c700 x1874531134720896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699155.603925:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134720896/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699155.603933:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699155.603936:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699155.603939:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134720896:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699155.603944:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721024:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699155.603957:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699155.603964:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699155/real 1787699155] req@ffff9d85b9d2ed80 x1874531134721024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699155.603971:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134721024/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699155.603978:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699155.603981:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699155.603984:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721024:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699155.603989:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699155.604006:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699155.604016:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699155/real 1787699155] req@ffff9d85b9d2e680 x1874531134721152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699155.604028:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134721152/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699155.604033:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699155.604035:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699155.604037:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721152:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699156.180042:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699157.202970:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699158.227065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699159.251955:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699160.276183:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699160.723487:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699160.723498:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699160.723502:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699160.723512:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7062 00000100:00080000:3.0:1787699160.723521:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699160.723528:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699160.723562:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134721280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699160.723594:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699160.723598:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699160.723600:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699160.723604:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7062 00000100:00080000:3.0:1787699160.723608:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699160.723612:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699160.723622:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699160.723635:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134721408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699160.723645:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699160.723648:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699160.723650:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699160.723654:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7062 00000100:00080000:3.0:1787699160.723656:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699160.723660:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699160.723664:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699160.723672:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134721536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699160.723680:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699160.723683:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699160.723685:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699160.723688:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7062 00000100:00080000:3.0:1787699160.723690:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699160.723694:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699160.723697:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699160.723704:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134721664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699160.723712:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699160.723715:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699160.723716:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699160.723719:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7062 00000100:00080000:3.0:1787699160.723722:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699160.723726:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699160.723729:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699160.723736:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134721792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699160.723743:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699160.723746:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699160.723747:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699160.723750:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7062 00000100:00080000:3.0:1787699160.723753:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699160.723756:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699160.723760:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699160.723769:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134721920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699160.723777:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699160.723779:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699160.723781:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699160.723784:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7062 00000100:00080000:3.0:1787699160.723787:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699160.723790:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699160.723794:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699160.723801:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134722048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699160.723895:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134721280:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699160.723950:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699160.723984:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699160/real 1787699160] req@ffff9d85c0fb1c00 x1874531134721280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699160.724000:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134721280/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699160.724011:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699160.724014:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699160.724018:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134721280:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699160.724026:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134721408:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699160.724044:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699160.724056:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699160/real 1787699160] req@ffff9d85c0fb0a80 x1874531134721408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699160.724069:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134721408/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699160.724078:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699160.724080:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699160.724084:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134721408:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699160.724090:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134721536:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699160.724109:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699160.724119:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699160/real 1787699160] req@ffff9d85c0fb3b80 x1874531134721536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699160.724130:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134721536/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699160.724138:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699160.724142:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699160.724145:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134721536:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699160.724150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134721664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699160.724168:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699160.724179:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699160/real 1787699160] req@ffff9d85c0fb0380 x1874531134721664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699160.724191:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134721664/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699160.725473:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699160.725493:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699160.725498:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134721664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699160.725522:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721792:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699160.725599:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699160.725615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699160/real 1787699160] req@ffff9d85c0fb2d80 x1874531134721792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699160.725629:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134721792/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699160.725638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699160.725641:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699160.725645:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721792:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699160.725651:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721920:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699160.725668:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699160.725678:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699160/real 1787699160] req@ffff9d85c0fb3800 x1874531134721920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699160.725691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134721920/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699160.725699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699160.725703:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699160.725707:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134721920:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699160.725714:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699160.725733:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699160.725744:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699160/real 1787699160] req@ffff9d85c0fb0000 x1874531134722048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699160.725756:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134722048/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699160.725764:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699160.725767:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699160.725770:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722048:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699161.299209:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699162.325222:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699163.347015:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699164.371197:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699165.395087:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699165.843003:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699165.843011:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699165.843015:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699165.843023:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7067 00000100:00080000:0.0:1787699165.843033:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699165.843039:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699165.843073:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134722176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699165.843104:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699165.843116:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699165.843118:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699165.843122:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7067 00000100:00080000:0.0:1787699165.843125:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699165.843129:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699165.843138:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699165.843151:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134722304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699165.843164:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134722176:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699165.843221:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699165.843252:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699165/real 1787699165] req@ffff9d85b9d0bb80 x1874531134722176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699165.843268:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134722176/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699165.843278:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699165.843282:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699165.843286:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134722176:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787699165.843303:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787699165.843308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134722304:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787699165.843309:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699165.843314:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699165.843350:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7067 00000100:00080000:0.0:1787699165.843356:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699165.843361:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699165.843367:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699165.843367:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699165.843383:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134722432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699165.843385:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699165/real 1787699165] req@ffff9d85b9d0ad80 x1874531134722304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699165.843407:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787699165.843412:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134722304/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699165.843413:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699165.843416:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699165.843424:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7067 00000100:00080000:0.0:1787699165.843430:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699165.843430:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699165.843436:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787699165.843438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699165.843443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134722304:192.168.201.137@tcp:38:kworker.0 00000008:00100000:0.0:1787699165.843452:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699165.843454:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134722432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699165.843468:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134722560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787699165.843476:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787699165.843484:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787699165.843489:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699165/real 1787699165] req@ffff9d85b9d08a80 x1874531134722432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699165.843490:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699165.843494:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699165.843502:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7067 00000100:00080000:0.0:1787699165.843507:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787699165.843511:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134722432/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699165.843512:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699165.843520:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787699165.843526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699165.843532:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699165.843533:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134722688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699165.843539:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134722432:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699165.843554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134722560:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787699165.843564:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699165.843568:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699165.843570:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699165.843575:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7067 00000800:00000100:1.0:1787699165.843580:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787699165.843582:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699165.843588:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699165.843593:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699165.843596:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699165/real 1787699165] req@ffff9d85b9d08000 x1874531134722560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699165.843606:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134722816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699165.843612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134722560/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699165.843620:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699165.843628:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699165.843631:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699165.843631:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699165.843637:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00080000:0.0:1787699165.843640:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7067 00000100:00100000:1.0:1787699165.843643:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134722560:192.168.201.137@tcp:8:kworker.0 00000100:00080000:0.0:1787699165.843645:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699165.843651:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699165.843655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722688:192.168.201.137@tcp:38:kworker.0 00000008:00100000:0.0:1787699165.843657:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699165.843672:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134722944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787699165.843677:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699165.843688:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699165/real 1787699165] req@ffff9d85b9d08e00 x1874531134722688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699165.843701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134722688/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699165.843706:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699165.843709:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699165.843711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722688:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699165.843720:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722816:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699165.843737:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699165.843746:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699165/real 1787699165] req@ffff9d85b9d0b480 x1874531134722816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699165.843758:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134722816/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699165.843763:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699165.843765:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699165.843767:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722816:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699165.843775:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699165.843787:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699165.843793:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699165/real 1787699165] req@ffff9d85b9d0a680 x1874531134722944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699165.843799:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134722944/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699165.843807:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699165.843809:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699165.843811:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134722944:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699166.419063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699167.443193:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699168.467051:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699169.490959:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699170.515845:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699170.963014:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699170.963055:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699170.963061:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699170.963074:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7072 00000100:00080000:0.0:1787699170.963088:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699170.963097:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699170.963147:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134723072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699170.963186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699170.963190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699170.963192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699170.963197:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7072 00000100:00080000:0.0:1787699170.963201:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699170.963205:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699170.963215:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699170.963229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134723200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699170.963238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699170.963242:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699170.963244:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699170.963248:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7072 00000100:00080000:0.0:1787699170.963253:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699170.963258:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699170.963264:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699170.963278:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134723328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699170.963293:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787699170.963294:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134723072:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787699170.963308:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699170.963312:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699170.963316:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7072 00000100:00080000:0.0:1787699170.963320:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699170.963324:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699170.963330:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699170.963347:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134723456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699170.963355:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699170.963359:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699170.963360:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699170.963364:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7072 00000100:00080000:0.0:1787699170.963369:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699170.963374:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699170.963378:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699170.963382:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699170.963396:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134723584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699170.963412:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699170.963416:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699170.963419:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699170.963425:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7072 00000100:00080000:0.0:1787699170.963429:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787699170.963430:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699170/real 1787699170] req@ffff9d85b9d09c00 x1874531134723072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699170.963438:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699170.963449:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699170.963452:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134723072/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699170.963465:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787699170.963471:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134723712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699170.963472:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699170.963480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134723072:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699170.963496:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134723200:192.168.201.137@tcp:38:kworker.0 00000100:00080000:0.0:1787699170.963507:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699170.963512:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699170.963514:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699170.963520:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7072 00000100:00080000:0.0:1787699170.963526:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787699170.963533:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:0.0:1787699170.963536:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699170.963544:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699170.963556:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699170/real 1787699170] req@ffff9d85b9d08700 x1874531134723200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699170.963563:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134723840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699170.963592:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134723200/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699170.963602:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699170.963605:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699170.963608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134723200:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699170.963636:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134723328:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699170.963654:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699170.963661:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699170/real 1787699170] req@ffff9d85b9d09500 x1874531134723328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699170.963667:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134723328/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699170.963672:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699170.963674:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699170.963676:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134723328:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699170.963679:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134723456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699170.963687:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699170.963692:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699170/real 1787699170] req@ffff9d85b9d09180 x1874531134723456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699170.963698:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134723456/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699170.963703:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699170.963705:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699170.963706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134723456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699170.963711:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134723584:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699170.963728:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699170.963738:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699170/real 1787699170] req@ffff9d85b9d0a300 x1874531134723584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699170.963749:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134723584/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699170.963756:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699170.963759:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699170.963762:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134723584:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699170.963768:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134723712:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699170.963786:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699170.963797:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699170/real 1787699170] req@ffff9d85b9d09f80 x1874531134723712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699170.963808:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134723712/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699170.963815:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699170.963818:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699170.963821:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134723712:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699170.963827:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134723840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699170.963844:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699170.963889:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699170/real 1787699170] req@ffff9d85b9d0b800 x1874531134723840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699170.963896:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134723840/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699170.963902:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699170.963904:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699170.963905:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134723840:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699171.539072:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699172.563056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699173.587211:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699174.611086:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699175.634985:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699176.083074:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699176.083084:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699176.083087:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699176.083098:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7077 00000100:00080000:0.0:1787699176.083108:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699176.083115:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699176.083152:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134723968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699176.083179:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699176.083182:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699176.083184:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699176.083188:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7077 00000100:00080000:0.0:1787699176.083191:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699176.083194:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699176.083203:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699176.083213:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134724096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699176.083221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699176.083223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699176.083225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699176.083228:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7077 00000100:00080000:0.0:1787699176.083231:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699176.083234:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699176.083238:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699176.083246:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134724224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699176.083253:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699176.083257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699176.083258:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699176.083261:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7077 00000100:00080000:0.0:1787699176.083264:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699176.083267:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699176.083271:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699176.083279:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134724352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699176.083287:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699176.083290:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699176.083291:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699176.083294:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7077 00000100:00080000:0.0:1787699176.083297:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699176.083300:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699176.083304:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699176.083312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134724480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699176.083318:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699176.083322:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699176.083323:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699176.083326:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7077 00000100:00080000:0.0:1787699176.083329:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699176.083332:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699176.083336:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699176.083344:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134724608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699176.083352:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699176.083354:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699176.083356:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699176.083359:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7077 00000100:00080000:0.0:1787699176.083362:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699176.083365:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699176.083369:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699176.083378:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134724736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699176.083994:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134723968:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699176.084093:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699176.084117:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699176.084134:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699176.084149:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134724480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699176.084164:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134724608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699176.084180:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134724736:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699176.659042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699177.683086:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699178.707124:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787699179.155174:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:2.0:1787699179.155251:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787699179.155273:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699179.155321:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699179.155333:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699179.155342:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699179.155353:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699179.155363:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:2.0:1787699179.155373:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787699179.155438:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699176/real 1787699179] req@ffff9d85b9d0b800 x1874531134723968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699179.155479:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134723968/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699179.155493:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699179.155499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699179.155503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134723968:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699179.155512:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699176/real 1787699179] req@ffff9d85b9d09f80 x1874531134724096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699179.155528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134724096/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699179.155536:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699179.155540:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699179.155543:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724096:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699179.155550:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699176/real 1787699179] req@ffff9d85b9d0a300 x1874531134724224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699179.155562:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134724224/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699179.155569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699179.155573:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699179.155577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724224:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699179.155584:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699176/real 1787699179] req@ffff9d85b9d09180 x1874531134724352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699179.155596:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134724352/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699179.155604:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699179.155614:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699179.155618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699179.155626:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699176/real 1787699179] req@ffff9d85b9d09500 x1874531134724480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699179.155640:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134724480/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699179.155647:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699179.155650:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699179.155653:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134724480:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699179.155661:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699176/real 1787699179] req@ffff9d85b9d08700 x1874531134724608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699179.155674:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134724608/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699179.155681:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699179.155684:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699179.155687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134724608:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699179.155694:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699176/real 1787699179] req@ffff9d85b9d09c00 x1874531134724736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699179.155703:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134724736/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699179.155712:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699179.155715:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699179.155718:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134724736:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699179.731058:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699180.757514:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699181.206504:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699181.206519:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699181.206523:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699181.206534:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7082 00000100:00080000:3.0:1787699181.206547:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699181.206556:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699181.206619:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134724864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699181.206668:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699181.206673:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699181.206674:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699181.206679:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7082 00000100:00080000:3.0:1787699181.206684:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699181.206688:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699181.206699:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699181.206712:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134724992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699181.206721:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699181.206724:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699181.206725:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699181.206730:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7082 00000100:00080000:3.0:1787699181.206733:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699181.206736:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699181.206742:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699181.206751:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134725120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699181.206758:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699181.206761:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699181.206762:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699181.206765:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7082 00000100:00080000:3.0:1787699181.206768:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699181.206771:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699181.206776:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699181.207095:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134725248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699181.207106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699181.207109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699181.207110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699181.207114:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7082 00000100:00080000:3.0:1787699181.207118:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699181.207122:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699181.207127:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699181.207137:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134725376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699181.207146:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699181.207150:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699181.207151:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699181.207155:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7082 00000100:00080000:3.0:1787699181.207157:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699181.207161:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699181.207165:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699181.207174:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134725504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699181.207182:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699181.207186:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699181.207188:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699181.207191:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7082 00000100:00080000:3.0:1787699181.207194:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699181.207198:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699181.207203:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699181.207211:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134725632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699181.207327:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134724864:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699181.207399:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699181.207448:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699181/real 1787699181] req@ffff9d85c0fb2a00 x1874531134724864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699181.207465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134724864/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699181.207476:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699181.207480:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699181.207484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134724864:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699181.207491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724992:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699181.207511:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699181.207523:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699181/real 1787699181] req@ffff9d85c0fb2680 x1874531134724992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699181.207535:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134724992/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699181.207542:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699181.207545:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699181.207548:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134724992:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699181.207554:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134725120:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699181.207574:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699181.207585:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699181/real 1787699181] req@ffff9d85c0fb0e00 x1874531134725120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699181.207600:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134725120/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699181.207607:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699181.207609:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699181.207612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134725120:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699181.207617:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134725248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699181.207635:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699181.207711:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699181/real 1787699181] req@ffff9d85c0fb0000 x1874531134725248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699181.207722:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134725248/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699181.207730:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699181.207734:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699181.207736:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134725248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699181.207742:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134725376:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699181.207763:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699181.207773:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699181/real 1787699181] req@ffff9d85c0fb3800 x1874531134725376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699181.208011:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134725376/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699181.208023:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699181.208026:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699181.208030:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134725376:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699181.208037:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134725504:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699181.208057:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699181.208068:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699181/real 1787699181] req@ffff9d85c0fb2d80 x1874531134725504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699181.208078:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134725504/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699181.208086:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699181.208088:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699181.208091:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134725504:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699181.208097:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134725632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699181.208115:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699181.208125:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699181/real 1787699181] req@ffff9d85c0fb0380 x1874531134725632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699181.208136:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134725632/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699181.208144:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699181.208147:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699181.208150:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134725632:192.168.201.137@tcp:8:kworker.0 00010000:00010000:1.0:1787699181.779820:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699182.803999:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699183.827031:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699184.851037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699185.875029:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699186.322993:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699186.323003:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699186.323008:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699186.323018:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7088 00000100:00080000:3.0:1787699186.323029:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699186.323036:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699186.323076:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134725760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699186.323110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699186.323114:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699186.323116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699186.323120:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7088 00000100:00080000:3.0:1787699186.323124:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699186.323128:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699186.323138:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699186.323151:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134725888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699186.323161:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699186.323164:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699186.323166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699186.323169:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7088 00000100:00080000:3.0:1787699186.323172:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699186.323175:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699186.323180:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699186.323190:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134726016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699186.323199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699186.323202:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699186.323204:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699186.323207:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7088 00000100:00080000:3.0:1787699186.323210:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699186.323213:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699186.323218:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699186.323227:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134726144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699186.323236:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699186.323239:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699186.323240:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699186.323243:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7088 00000100:00080000:3.0:1787699186.323246:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699186.323251:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699186.323255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699186.323262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134726272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699186.323270:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699186.323273:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699186.323274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699186.323277:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7088 00000100:00080000:3.0:1787699186.323280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699186.323283:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699186.323287:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699186.323304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134726400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699186.323312:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699186.323315:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699186.323316:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699186.323319:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7088 00000100:00080000:3.0:1787699186.323322:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699186.323325:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699186.323329:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699186.323338:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134726528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699186.323436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134725760:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699186.323570:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699186.323619:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699186/real 1787699186] req@ffff9d85c0fb3b80 x1874531134725760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699186.323634:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134725760/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699186.323644:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699186.323647:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699186.323651:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134725760:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699186.323658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134725888:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699186.323677:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699186.323687:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699186/real 1787699186] req@ffff9d85c0fb0a80 x1874531134725888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699186.323697:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134725888/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699186.323716:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699186.323719:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699186.323722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134725888:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699186.323728:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726016:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699186.323748:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699186.323774:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699186/real 1787699186] req@ffff9d85c0fb1c00 x1874531134726016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699186.323787:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134726016/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699186.323794:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699186.323797:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699186.323800:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726016:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699186.323806:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699186.323834:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699186.323844:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699186/real 1787699186] req@ffff9d85c0fb3480 x1874531134726144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699186.323881:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134726144/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699186.323888:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699186.323891:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699186.323894:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699186.323899:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134726272:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699186.323915:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699186.323924:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699186/real 1787699186] req@ffff9d85c0fb2300 x1874531134726272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699186.323940:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134726272/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699186.323947:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699186.323950:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699186.323953:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134726272:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699186.323958:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134726400:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699186.323971:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699186.323978:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699186/real 1787699186] req@ffff9d85c0fb1f80 x1874531134726400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699186.323988:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134726400/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699186.323995:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699186.323998:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699186.324001:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134726400:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699186.324006:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134726528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699186.324020:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699186.324029:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699186/real 1787699186] req@ffff9d85c0fb0700 x1874531134726528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699186.324038:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134726528/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699186.324050:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699186.324052:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699186.324055:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134726528:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699186.899045:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699187.923189:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699188.947119:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699189.971176:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699190.995014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699191.444397:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699191.445650:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699191.445654:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699191.445696:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7093 00000100:00080000:0.0:1787699191.445753:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699191.445772:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699191.446131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134726656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699191.446177:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699191.446180:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699191.446182:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699191.446187:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7093 00000100:00080000:0.0:1787699191.446191:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699191.446195:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699191.446213:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699191.446227:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134726784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699191.446236:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699191.446239:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699191.446241:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699191.446251:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7093 00000100:00080000:0.0:1787699191.446254:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699191.446256:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699191.446260:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699191.446269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134726912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699191.446274:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699191.446276:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699191.446277:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699191.446278:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7093 00000100:00080000:0.0:1787699191.446280:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699191.446282:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699191.446284:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699191.446289:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134727040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699191.446298:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699191.446300:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699191.446301:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699191.446302:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7093 00000100:00080000:0.0:1787699191.446304:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699191.446306:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699191.446308:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699191.446312:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134727168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699191.446317:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699191.446324:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699191.446325:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699191.446327:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7093 00000100:00080000:0.0:1787699191.446329:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699191.446331:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699191.446333:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699191.446339:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134727296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699191.446343:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699191.446346:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699191.446346:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699191.446348:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7093 00000100:00080000:0.0:1787699191.446350:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699191.446352:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699191.446354:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699191.446359:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134727424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699191.450026:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134726656:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699191.452222:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699191.452289:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699191/real 1787699191] req@ffff9d85b9d0a680 x1874531134726656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699191.452308:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134726656/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699191.452319:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699191.452324:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699191.452328:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134726656:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699191.452338:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726784:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699191.452360:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699191.452372:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699191/real 1787699191] req@ffff9d85b9d0b480 x1874531134726784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699191.452383:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134726784/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699191.452391:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699191.452395:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699191.452399:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726784:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699191.452406:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726912:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699191.452424:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699191.452435:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699191/real 1787699191] req@ffff9d85b9d08e00 x1874531134726912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699191.452446:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134726912/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699191.452453:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699191.452456:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699191.452460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134726912:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699191.452465:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699191.452481:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699191.452491:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699191/real 1787699191] req@ffff9d85b9d08000 x1874531134727040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699191.452501:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134727040/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699191.452508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699191.452512:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699191.452574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699191.452581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134727168:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699191.452602:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699191.452613:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699191/real 1787699191] req@ffff9d85b9d08a80 x1874531134727168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699191.452624:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134727168/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699191.452632:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699191.452635:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699191.452638:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134727168:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699191.452643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134727296:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699191.452661:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699191.452783:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699191/real 1787699191] req@ffff9d85b9d0ad80 x1874531134727296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699191.452800:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134727296/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699191.452808:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699191.452811:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699191.452997:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134727296:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699191.453005:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134727424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699191.453028:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699191.453040:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699191/real 1787699191] req@ffff9d85b9d0bb80 x1874531134727424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699191.453051:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134727424/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699191.453060:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699191.453063:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699191.453067:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134727424:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699192.019105:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699193.043065:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699194.067049:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699195.091014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699196.115013:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699196.562985:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699196.562994:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699196.562997:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699196.563007:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7098 00000100:00080000:0.0:1787699196.563018:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699196.563024:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699196.563061:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0bb80 x1874531134727552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699196.563090:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699196.563093:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699196.563095:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699196.563100:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7098 00000100:00080000:0.0:1787699196.563103:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699196.563107:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699196.563115:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699196.563128:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0ad80 x1874531134727680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699196.563138:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699196.563144:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699196.563146:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699196.563150:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7098 00000100:00080000:0.0:1787699196.563153:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699196.563157:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699196.563163:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699196.563173:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08a80 x1874531134727808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699196.563180:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134727552:192.168.201.137@tcp:250:kworker.0 00000100:00080000:0.0:1787699196.563186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699196.563189:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699196.563191:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699196.563193:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7098 00000100:00080000:0.0:1787699196.563196:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699196.563198:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699196.563202:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699196.563211:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08000 x1874531134727936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699196.563217:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699196.563219:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699196.563220:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699196.563222:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7098 00000100:00080000:0.0:1787699196.563233:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699196.563235:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699196.563239:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699196.563244:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699196.563256:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08e00 x1874531134728064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699196.563266:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699196.563269:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699196.563271:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699196.563275:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7098 00000100:00080000:0.0:1787699196.563279:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699196.563283:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699196.563288:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699196/real 1787699196] req@ffff9d85b9d0bb80 x1874531134727552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699196.563291:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699196.563304:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b480 x1874531134728192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699196.563311:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699196.563314:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699196.563316:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699196.563319:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7098 00000100:00080000:0.0:1787699196.563321:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699196.563324:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699196.563328:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699196.563337:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a680 x1874531134728320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699196.563459:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0bb80 x1874531134727552/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699196.563470:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699196.563473:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699196.563477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134727552:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699196.563484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727680:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699196.563519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699196.563532:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699196/real 1787699196] req@ffff9d85b9d0ad80 x1874531134727680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699196.563544:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0ad80 x1874531134727680/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699196.563552:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699196.563555:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699196.563558:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727680:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699196.563581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727808:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699196.563642:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699196.563666:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699196/real 1787699196] req@ffff9d85b9d08a80 x1874531134727808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699196.563677:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08a80 x1874531134727808/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699196.563685:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699196.563688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699196.563691:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727808:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699196.563697:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699196.563716:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699196.563727:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699196/real 1787699196] req@ffff9d85b9d08000 x1874531134727936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699196.563738:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08000 x1874531134727936/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699196.563746:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699196.563749:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699196.563752:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134727936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699196.563757:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728064:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699196.563774:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699196.563785:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699196/real 1787699196] req@ffff9d85b9d08e00 x1874531134728064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699196.563795:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08e00 x1874531134728064/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699196.563803:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699196.563805:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699196.563808:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728064:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699196.563814:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728192:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699196.563829:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699196.563838:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699196/real 1787699196] req@ffff9d85b9d0b480 x1874531134728192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699196.563848:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b480 x1874531134728192/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699196.563876:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699196.563879:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699196.563882:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728192:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699196.563887:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699196.563911:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699196.563922:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699196/real 1787699196] req@ffff9d85b9d0a680 x1874531134728320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699196.563933:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a680 x1874531134728320/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699196.563940:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699196.563943:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699196.563946:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728320:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699197.139015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699198.163287:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699199.187659:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699200.211041:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699201.235124:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699201.683445:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699201.683456:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699201.683460:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699201.683470:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7103 00000100:00080000:3.0:1787699201.683482:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699201.683489:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699201.683539:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134728448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699201.683570:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699201.683574:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699201.683576:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699201.683580:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7103 00000100:00080000:3.0:1787699201.683584:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699201.683587:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699201.683596:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699201.683609:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134728576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699201.683618:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699201.683621:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699201.683623:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699201.683627:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7103 00000100:00080000:3.0:1787699201.683630:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699201.683633:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699201.683639:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699201.683649:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134728704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699201.683658:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699201.683661:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699201.683663:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699201.683666:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7103 00000100:00080000:3.0:1787699201.683669:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699201.683672:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699201.683677:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699201.683690:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134728832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699201.683698:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699201.683701:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699201.683703:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699201.683706:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7103 00000100:00080000:3.0:1787699201.683709:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699201.683713:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699201.683717:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699201.683726:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134728960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699201.683734:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699201.683737:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699201.683738:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699201.683742:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7103 00000100:00080000:3.0:1787699201.683745:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699201.683749:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699201.683753:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699201.683762:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134729088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699201.683771:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699201.683774:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699201.683776:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699201.683779:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7103 00000100:00080000:3.0:1787699201.683782:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699201.683785:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699201.683789:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699201.683799:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134729216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699201.683949:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134728448:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699201.684010:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699201.684105:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699201/real 1787699201] req@ffff9d85c0fb1500 x1874531134728448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699201.684123:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134728448/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699201.684132:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699201.684136:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699201.684140:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134728448:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699201.684147:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134728576:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699201.684170:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699201.684194:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699201/real 1787699201] req@ffff9d85c0fb1880 x1874531134728576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699201.684206:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134728576/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699201.684214:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699201.684217:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699201.684221:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134728576:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699201.684227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134728704:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699201.684248:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699201.684260:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699201/real 1787699201] req@ffff9d85c0fb3100 x1874531134728704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699201.684272:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134728704/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699201.684280:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699201.684283:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699201.684287:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134728704:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699201.684293:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134728832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699201.684318:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699201.684330:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699201/real 1787699201] req@ffff9d85c0fb0700 x1874531134728832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699201.684409:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134728832/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699201.684419:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699201.684422:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699201.684425:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134728832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699201.684431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728960:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699201.684452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699201.684464:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699201/real 1787699201] req@ffff9d85c0fb1f80 x1874531134728960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699201.684475:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134728960/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699201.684483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699201.684486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699201.684489:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134728960:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699201.684495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729088:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699201.684528:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699201.684539:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699201/real 1787699201] req@ffff9d85c0fb2300 x1874531134729088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699201.684550:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134729088/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699201.684558:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699201.684561:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699201.684564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729088:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699201.684569:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699201.684588:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699201.684598:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699201/real 1787699201] req@ffff9d85c0fb3480 x1874531134729216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699201.684610:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134729216/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699201.684618:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699201.684621:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699201.684624:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729216:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699202.262484:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699203.283088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699204.307334:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699205.332153:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699206.356865:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699206.803030:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699206.803039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699206.803044:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699206.803055:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7108 00000100:00080000:3.0:1787699206.803065:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699206.803490:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699206.803530:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134729344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699206.803739:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134729344:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699206.803798:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699206.803829:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699206.803833:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699206.803835:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699206.803840:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7108 00000100:00080000:3.0:1787699206.803844:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699206.803848:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699206.804147:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699206.804163:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134729472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699206.804327:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699206.804331:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699206.804333:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699206.804337:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7108 00000100:00080000:3.0:1787699206.804341:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699206.804344:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699206.804350:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699206.804362:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134729600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699206.804371:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699206.804375:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699206.804377:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699206.804380:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7108 00000100:00080000:3.0:1787699206.804383:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699206.804387:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699206.804836:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699206.804884:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134729728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699206.804895:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699206.804899:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699206.804901:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699206.804904:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7108 00000100:00080000:3.0:1787699206.804909:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699206.804914:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699206.804918:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699206.804928:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134729856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699206.804936:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699206.804939:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699206.804940:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699206.804943:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7108 00000100:00080000:3.0:1787699206.804946:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699206.804949:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699206.804953:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699206.804961:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134729984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699206.804968:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699206.804970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699206.804972:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699206.804974:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7108 00000100:00080000:3.0:1787699206.804977:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699206.804980:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699206.804983:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699206.804990:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134730112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699206.805145:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699206/real 1787699206] req@ffff9d85c0fb1c00 x1874531134729344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699206.805164:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134729344/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699206.805173:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699206.805177:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699206.805181:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134729344:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699206.805205:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134729472:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699206.805238:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699206.805253:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699206/real 1787699206] req@ffff9d85c0fb0a80 x1874531134729472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699206.805263:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134729472/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699206.805270:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699206.805273:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699206.805276:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134729472:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699206.805282:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134729600:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699206.805300:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699206.805310:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699206/real 1787699206] req@ffff9d85c0fb3b80 x1874531134729600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699206.805320:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134729600/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699206.805326:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699206.805329:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699206.805331:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134729600:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699206.805337:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134729728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699206.805354:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699206.805364:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699206/real 1787699206] req@ffff9d85c0fb0380 x1874531134729728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699206.805373:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134729728/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699206.805380:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699206.805382:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699206.805385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134729728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699206.805402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729856:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699206.806555:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699206.806580:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699206/real 1787699206] req@ffff9d85c0fb2d80 x1874531134729856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699206.806596:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134729856/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699206.806605:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699206.806608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699206.806612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729856:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699206.806619:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729984:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699206.806669:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699206.806680:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699206/real 1787699206] req@ffff9d85c0fb3800 x1874531134729984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699206.806701:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134729984/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699206.806709:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699206.806713:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699206.806716:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134729984:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699206.806723:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134730112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699206.806741:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699206.806752:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699206/real 1787699206] req@ffff9d85c0fb0000 x1874531134730112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699206.806762:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134730112/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699206.806770:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699206.806773:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699206.806776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134730112:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699207.380571:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699208.403448:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699209.428240:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699210.451171:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699211.475818:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699211.923023:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699211.923037:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699211.923042:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699211.923055:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7113 00000100:00080000:2.0:1787699211.923069:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699211.923078:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699211.923129:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134730240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699211.923226:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134730240:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699211.923300:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787699211.923338:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699211.923342:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699211.923343:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699211.923348:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7113 00000100:00080000:2.0:1787699211.923351:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699211.923355:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699211.923367:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699211.923380:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134730368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699211.923604:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699211/real 1787699211] req@ffff9d85babd6300 x1874531134730240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699211.923625:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134730240/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699211.923648:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699211.923653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699211.923657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134730240:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699211.923683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134730368:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699211.923705:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699211.923717:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699211/real 1787699211] req@ffff9d85babd4380 x1874531134730368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699211.923727:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134730368/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699211.923734:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699211.923737:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699211.923740:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134730368:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787699211.924897:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699211.924907:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699211.924910:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699211.924917:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7113 00000100:00080000:2.0:1787699211.924925:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699211.924931:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699211.924941:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699211.924965:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134730496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699211.925467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134730496:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699211.925519:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787699211.925546:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699211.925550:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699211.925553:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699211.925558:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7113 00000100:00080000:2.0:1787699211.925563:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699211.925567:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699211.925573:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699211.925634:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134730624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699211.925679:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699211/real 1787699211] req@ffff9d85babd6680 x1874531134730496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699211.925697:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134730496/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699211.925705:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699211.925708:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699211.925712:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134730496:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699211.925731:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134730624:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787699211.925818:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699211.925823:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699211.925825:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699211.925830:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7113 00000100:00080000:2.0:1787699211.925834:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699211.925839:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699211.925846:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699211.928674:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699211.928730:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699211/real 1787699211] req@ffff9d85babd6a00 x1874531134730624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699211.928767:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134730624/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699211.928780:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699211.928785:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699211.928789:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134730624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:2.0:1787699211.928927:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134730752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699211.928966:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699211.928971:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699211.928973:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699211.928978:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7113 00000100:00080000:2.0:1787699211.928996:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699211.929003:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699211.929014:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699211.929026:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134730880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699211.929040:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699211.929043:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699211.929044:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699211.929048:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7113 00000100:00080000:2.0:1787699211.929051:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699211.929055:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699211.929059:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699211.929069:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134731008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699211.929150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134730752:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699211.929213:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699211.929226:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699211/real 1787699211] req@ffff9d85babd4e00 x1874531134730752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699211.929239:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134730752/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699211.929247:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699211.929250:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699211.929253:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134730752:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699211.929259:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134730880:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699211.929277:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699211.929286:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699211/real 1787699211] req@ffff9d85babd7b80 x1874531134730880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699211.929296:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134730880/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699211.929304:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699211.929307:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699211.929311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134730880:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699211.929316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699211.929347:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699211.929358:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699211/real 1787699211] req@ffff9d85babd5f80 x1874531134731008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699211.929369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134731008/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699211.929383:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699211.929387:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699211.929390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731008:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699212.499224:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699213.523101:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699214.547117:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699215.572332:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699216.595895:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699217.043028:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699217.043042:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699217.043048:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699217.043062:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7118 00000100:00080000:2.0:1787699217.043090:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699217.043099:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699217.043165:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134731136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699217.043318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134731136:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699217.043410:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:2.0:1787699217.043451:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699217.043455:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699217.043458:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699217.043463:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7118 00000100:00080000:2.0:1787699217.043467:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699217.043471:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699217.043482:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699217.043498:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134731264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699217.046551:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699217/real 1787699217] req@ffff9d85babd4700 x1874531134731136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699217.046589:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134731136/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699217.046603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699217.046608:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699217.046613:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134731136:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699217.046655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134731264:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699217.046945:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699217.046966:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699217/real 1787699217] req@ffff9d85babd5180 x1874531134731264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699217.046980:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134731264/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699217.046989:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699217.046992:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699217.046996:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134731264:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787699217.051920:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699217.051934:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699217.051939:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699217.051952:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7118 00000100:00080000:2.0:1787699217.051966:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699217.051975:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699217.051991:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699217.052037:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134731392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699217.052055:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699217.052058:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699217.052060:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699217.052064:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7118 00000100:00080000:2.0:1787699217.052068:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699217.052072:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699217.052089:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699217.052103:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134731520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699217.052111:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699217.052115:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699217.052116:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699217.052120:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7118 00000100:00080000:2.0:1787699217.052123:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699217.052127:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699217.052133:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699217.052411:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134731648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699217.052422:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699217.052425:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699217.052427:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699217.052431:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7118 00000100:00080000:2.0:1787699217.052435:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699217.052439:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699217.052445:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699217.052459:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134731776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699217.052467:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699217.052470:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699217.052471:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699217.052475:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7118 00000100:00080000:2.0:1787699217.052479:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699217.052482:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699217.052486:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699217.052496:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134731904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699217.054932:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134731392:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699217.055031:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699217.055072:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699217/real 1787699217] req@ffff9d85babd4a80 x1874531134731392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699217.055108:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134731392/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699217.055120:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699217.055124:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699217.055128:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134731392:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699217.055136:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134731520:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699217.056304:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699217.056329:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699217/real 1787699217] req@ffff9d85babd7100 x1874531134731520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699217.056345:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134731520/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699217.056354:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699217.056359:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699217.056363:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134731520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699217.056372:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731648:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699217.056403:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699217.056415:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699217/real 1787699217] req@ffff9d85babd5180 x1874531134731648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699217.056427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134731648/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699217.056434:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699217.056438:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699217.056441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731648:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699217.056448:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731776:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699217.056467:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699217.056478:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699217/real 1787699217] req@ffff9d85babd4700 x1874531134731776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699217.056488:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134731776/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699217.056496:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699217.056499:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699217.056502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731776:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699217.056508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731904:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699217.056525:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699217.056535:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699217/real 1787699217] req@ffff9d85babd5f80 x1874531134731904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699217.056545:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134731904/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699217.056560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699217.056563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699217.056566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134731904:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699217.624112:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699218.643474:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699219.667886:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699220.691098:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699221.715083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699222.163999:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699222.164013:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699222.164018:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699222.164031:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7123 00000100:00080000:3.0:1787699222.164045:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699222.164054:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699222.165377:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134732032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699222.165498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134732032:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699222.165585:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699222.165747:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699222.165752:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699222.165754:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699222.165761:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7123 00000100:00080000:3.0:1787699222.165766:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699222.165772:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699222.165785:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699222.165804:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134732160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699222.165979:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699222/real 1787699222] req@ffff9d85c0fb0e00 x1874531134732032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699222.165996:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134732032/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699222.166005:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699222.166008:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699222.166011:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134732032:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699222.166036:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134732160:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699222.166262:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699222.166277:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699222/real 1787699222] req@ffff9d85c0fb2680 x1874531134732160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699222.166290:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134732160/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699222.166296:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699222.166298:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699222.166300:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134732160:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699222.166566:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699222.166571:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699222.166573:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699222.166577:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7123 00000100:00080000:3.0:1787699222.166582:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699222.166585:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699222.166592:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699222.166697:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134732288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699222.166715:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699222.166719:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699222.166721:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699222.166725:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7123 00000100:00080000:3.0:1787699222.166729:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699222.166733:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699222.166737:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699222.166748:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134732288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787699222.166752:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134732416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699222.166762:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699222.166765:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699222.166766:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699222.166769:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7123 00000100:00080000:3.0:1787699222.166772:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699222.166774:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699222.166777:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699222.166785:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:3.0:1787699222.166786:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134732544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699222.166802:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699222/real 1787699222] req@ffff9d85c0fb2a00 x1874531134732288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699222.166815:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134732288/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699222.166823:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699222.166826:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699222.166829:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134732288:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699222.166842:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134732416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699222.166880:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699222.166893:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699222/real 1787699222] req@ffff9d85c0fb2680 x1874531134732416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699222.166904:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134732416/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699222.166913:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699222.166917:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699222.166920:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134732416:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787699222.167214:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699222.167219:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699222.167220:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699222.167226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7123 00000100:00080000:3.0:1787699222.167231:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699222.167236:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699222.167242:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134732544:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787699222.167243:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699222.167262:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134732672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787699222.167268:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699222.167280:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787699222.167283:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699222/real 1787699222] req@ffff9d85c0fb0e00 x1874531134732544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699222.167285:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699222.167289:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699222.167297:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7123 00000100:00100000:1.0:1787699222.167299:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134732544/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699222.167305:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699222.167308:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699222.167313:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00080000:1.0:1787699222.167314:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699222.167320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134732544:192.168.201.137@tcp:38:kworker.0 00000008:00100000:3.0:1787699222.167323:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699222.167331:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134732672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:3.0:1787699222.167339:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134732800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787699222.167348:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699222.167360:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699222/real 1787699222] req@ffff9d85c0fb0000 x1874531134732672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699222.167369:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134732672/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699222.167374:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699222.167377:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699222.167380:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134732672:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699222.167391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134732800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699222.167408:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699222.167418:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699222/real 1787699222] req@ffff9d85c0fb3800 x1874531134732800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699222.167429:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134732800/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699222.167555:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699222.167558:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699222.167560:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134732800:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699222.739873:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699223.763023:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699224.787285:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699225.811275:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699226.835022:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699227.286576:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699227.286586:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699227.286590:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699227.286601:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7129 00000100:00080000:2.0:1787699227.286612:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699227.286619:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699227.286660:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134732928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699227.286686:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699227.286694:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699227.286696:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699227.286700:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7129 00000100:00080000:2.0:1787699227.286704:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699227.286709:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699227.286720:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699227.286726:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134732928:192.168.201.137@tcp:250:kworker.0 00000100:00100000:2.0:1787699227.286736:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134733056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699227.286752:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699227.286755:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699227.286757:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699227.286760:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7129 00000100:00080000:2.0:1787699227.286763:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699227.286767:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699227.286773:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699227.286784:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134733184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699227.286793:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699227.286797:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699227.286798:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699227.286802:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7129 00000100:00080000:2.0:1787699227.286806:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699227.286810:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699227.286816:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699227.286821:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787699227.286832:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134733312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699227.286842:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699227.286845:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699227.286847:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787699227.286889:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699227/real 1787699227] req@ffff9d85babd7b80 x1874531134732928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699227.286895:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7129 00000100:00080000:2.0:1787699227.286902:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787699227.286914:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134732928/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699227.286925:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699227.286929:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699227.286933:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134732928:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699227.286954:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733056:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699227.286973:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699227.286983:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699227/real 1787699227] req@ffff9d85babd4e00 x1874531134733056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699227.286995:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134733056/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699227.287003:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699227.287006:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699227.287009:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733056:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699227.287015:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733184:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699227.287044:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699227.287054:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699227/real 1787699227] req@ffff9d85babd6a00 x1874531134733184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699227.287103:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134733184/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699227.287111:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699227.287114:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699227.287118:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733184:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699227.287123:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733312:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699227.287140:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699227.287148:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699227/real 1787699227] req@ffff9d85babd6680 x1874531134733312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699227.287160:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134733312/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699227.287168:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699227.287171:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699227.287174:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733312:192.168.201.137@tcp:8:kworker.0 00000100:00080000:2.0:1787699227.287267:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699227.287272:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699227.287285:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134733440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699227.287304:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699227.287308:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699227.287310:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699227.287313:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7129 00000100:00080000:2.0:1787699227.287318:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699227.287321:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699227.287325:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699227.287335:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134733568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699227.287346:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787699227.287347:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134733440:192.168.201.137@tcp:38:kworker.0 00000100:00080000:2.0:1787699227.287352:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699227.287355:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699227.287359:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7129 00000100:00080000:2.0:1787699227.287363:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699227.287367:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699227.287373:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787699227.287375:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787699227.287388:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134733696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699227.287392:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699227/real 1787699227] req@ffff9d85babd4380 x1874531134733440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699227.287415:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134733440/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699227.287426:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699227.287429:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699227.287432:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134733440:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699227.287442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134733568:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699227.287457:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699227.287465:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699227/real 1787699227] req@ffff9d85babd6300 x1874531134733568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699227.287476:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134733568/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699227.287483:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699227.287486:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699227.287490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134733568:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699227.287494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134733696:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699227.287506:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699227.287514:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699227/real 1787699227] req@ffff9d85babd7480 x1874531134733696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699227.287536:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134733696/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699227.287550:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699227.287553:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699227.287556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134733696:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699227.860234:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699228.883076:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699229.907061:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699230.935211:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699231.955042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699232.406067:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699232.406078:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699232.406083:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699232.406093:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7134 00000100:00080000:3.0:1787699232.406103:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699232.406110:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699232.406145:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134733824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699232.406174:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699232.406178:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699232.406180:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699232.406184:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7134 00000100:00080000:3.0:1787699232.406187:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699232.406190:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699232.406200:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699232.406210:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134733952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699232.406218:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699232.406221:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699232.406223:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699232.406226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7134 00000100:00080000:3.0:1787699232.406229:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787699232.406323:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134733824:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699232.406379:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699232.406427:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699232/real 1787699232] req@ffff9d85c0fb2d80 x1874531134733824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699232.406450:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134733824/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699232.406460:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699232.406463:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699232.406468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134733824:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699232.406474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733952:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699232.406551:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699232.406562:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699232/real 1787699232] req@ffff9d85c0fb0380 x1874531134733952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699232.406573:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134733952/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699232.406581:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699232.406584:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699232.406587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134733952:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699232.406683:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699232.406688:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699232.406699:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134734080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699232.406801:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734080:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699232.406821:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699232.406841:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699232.406845:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699232.406846:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699232.406875:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7134 00000100:00080000:3.0:1787699232.406879:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699232.406883:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699232.406887:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699232.406899:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134734208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699232.410126:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699232/real 1787699232] req@ffff9d85c0fb3b80 x1874531134734080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699232.410151:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134734080/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699232.410161:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699232.410166:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699232.410170:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734080:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699232.410195:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734208:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699232.410253:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699232.410269:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699232/real 1787699232] req@ffff9d85c0fb0a80 x1874531134734208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699232.410281:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134734208/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699232.410289:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699232.410292:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699232.410296:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734208:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787699232.410814:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699232.410819:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699232.410822:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699232.410827:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7134 00000100:00080000:3.0:1787699232.410833:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699232.410839:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699232.410847:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699232.410881:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134734336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699232.410934:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134734336:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699232.410957:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699232.410984:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699232.410988:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699232.410990:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699232.410994:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7134 00000100:00080000:3.0:1787699232.410998:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699232.411001:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699232.411006:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699232.411017:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134734464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699232.411097:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699232/real 1787699232] req@ffff9d85c0fb1c00 x1874531134734336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699232.411110:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134734336/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699232.411119:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699232.411122:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699232.411125:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134734336:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699232.411136:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134734464:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699232.411154:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699232.411162:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699232/real 1787699232] req@ffff9d85c0fb3480 x1874531134734464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699232.411172:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134734464/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699232.411180:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699232.411183:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699232.411186:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134734464:192.168.201.137@tcp:8:kworker.0 00000100:00080000:3.0:1787699232.411313:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699232.411317:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699232.411319:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699232.411323:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7134 00000100:00080000:3.0:1787699232.411328:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699232.411332:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699232.411337:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699232.411350:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134734592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699232.411392:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134734592:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699232.411424:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699232.411593:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699232/real 1787699232] req@ffff9d85c0fb2300 x1874531134734592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699232.411673:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134734592/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699232.411738:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699232.411743:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699232.411746:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134734592:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699232.979684:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699234.003126:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699235.027218:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699236.051988:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699237.075221:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699237.523153:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699237.523163:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699237.523167:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699237.523177:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7139 00000100:00080000:1.0:1787699237.523188:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699237.523194:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699237.523230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134734720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699237.523731:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134734720:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699237.523796:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699237.523832:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699237.523836:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699237.523838:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699237.523843:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7139 00000100:00080000:1.0:1787699237.523847:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699237.523907:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699237.523916:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699237.523931:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134734848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699237.533098:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699237/real 1787699237] req@ffff9d85b9d2d500 x1874531134734720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699237.533127:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134734720/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699237.533138:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699237.533142:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699237.533146:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134734720:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699237.533179:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734848:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699237.533235:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699237.533265:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699237/real 1787699237] req@ffff9d85b9d2dc00 x1874531134734848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699237.533277:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134734848/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699237.533285:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699237.533288:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699237.533292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734848:192.168.201.137@tcp:38:kworker.0 00000100:00080000:1.0:1787699237.534363:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699237.534370:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699237.534374:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699237.534380:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7139 00000100:00080000:1.0:1787699237.534387:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699237.534393:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699237.534404:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699237.534431:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134734976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699237.534445:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699237.534448:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699237.534450:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699237.534454:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7139 00000100:00080000:1.0:1787699237.534457:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699237.534461:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699237.534465:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699237.534594:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134735104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699237.534605:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699237.534608:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699237.534610:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699237.534614:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7139 00000100:00080000:1.0:1787699237.534617:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699237.534621:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699237.534627:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699237.534640:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134735232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699237.534649:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699237.534652:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699237.534653:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699237.534657:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7139 00000100:00080000:1.0:1787699237.534660:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699237.534664:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699237.534668:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699237.534678:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134735360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699237.534686:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699237.534689:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699237.534691:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699237.534694:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7139 00000100:00080000:1.0:1787699237.534698:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699237.534701:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699237.534705:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699237.534715:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134735488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699237.535375:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734976:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699237.535428:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699237.535449:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699237/real 1787699237] req@ffff9d85b9d2f480 x1874531134734976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699237.535465:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134734976/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699237.535684:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699237.535688:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699237.535692:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134734976:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699237.535699:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134735104:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699237.535722:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699237.535733:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699237/real 1787699237] req@ffff9d85b9d2ca80 x1874531134735104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699237.535745:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134735104/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699237.535753:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699237.535756:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699237.535759:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134735104:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699237.535765:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134735232:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699237.535784:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699237.535795:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699237/real 1787699237] req@ffff9d85b9d2f800 x1874531134735232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699237.535807:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134735232/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699237.535813:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699237.535816:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699237.535819:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134735232:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699237.535825:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134735360:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699237.535843:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699237.535873:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699237/real 1787699237] req@ffff9d85b9d2ce00 x1874531134735360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699237.536022:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134735360/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699237.536031:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699237.536034:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699237.536037:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134735360:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699237.536043:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134735488:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699237.536063:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699237.536074:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699237/real 1787699237] req@ffff9d85b9d2c000 x1874531134735488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699237.536084:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134735488/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699237.536091:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699237.536093:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699237.536096:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134735488:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699238.099571:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699239.123142:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699240.147652:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699241.171040:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699242.195191:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699242.643576:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699242.643590:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699242.643595:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699242.643609:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7144 00000100:00080000:1.0:1787699242.643624:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699242.643633:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699242.643744:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134735616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699242.643782:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699242.643786:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699242.643787:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699242.643792:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7144 00000100:00080000:1.0:1787699242.643796:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699242.643800:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699242.643842:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699242.644444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134735616:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699242.644718:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134735744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699242.644731:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699242.644734:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699242.644736:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699242.644739:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7144 00000100:00080000:1.0:1787699242.644743:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699242.644747:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699242.644753:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699242.644765:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134735872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699242.644793:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699242.644796:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699242.644798:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699242.644809:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7144 00000100:00080000:1.0:1787699242.644813:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699242.644817:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699242.644822:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699242.644833:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134736000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699242.644842:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699242.644846:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699242.644847:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699242.644867:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7144 00000100:00080000:1.0:1787699242.644871:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699242.644875:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699242.644880:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699242.644890:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134736128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699242.644898:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699242.644901:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699242.644903:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699242.644907:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7144 00000100:00080000:1.0:1787699242.644911:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699242.644916:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699242.644921:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699242.644931:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134736256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699242.644940:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699242.644943:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699242.644944:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699242.644948:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7144 00000100:00080000:1.0:1787699242.644951:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699242.644954:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699242.644958:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699242.644968:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134736384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699242.645135:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134735744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699242.645164:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134735872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699242.645181:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699242.645200:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134736128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699242.645222:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134736256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699242.645240:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134736384:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699243.219185:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699244.254373:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699245.268815:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787699245.715914:0:2350:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:3.0:1787699245.715973:0:2350:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787699245.715993:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787699245.720698:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699242/real 1787699245] req@ffff9d85b9d2fb80 x1874531134735616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699267 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699245.720747:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134735616/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699267 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699245.720773:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699245.720778:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699245.720782:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134735616:192.168.201.137@tcp:250:kworker.0 00000800:00000100:3.0:1787699245.721232:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699245.721517:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699245.721527:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699245.721537:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699245.721544:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699245.721553:0:2350:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:0.0:1787699245.721621:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699242/real 1787699245] req@ffff9d85b9d2c000 x1874531134735744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699245.721638:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134735744/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699245.721654:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699245.721659:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699245.721665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134735744:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699245.721675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699242/real 1787699245] req@ffff9d85b9d2ce00 x1874531134735872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699245.721692:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134735872/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699245.721700:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699245.721704:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699245.721707:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134735872:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699245.721718:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699242/real 1787699245] req@ffff9d85b9d2f800 x1874531134736000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699245.721730:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134736000/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699245.721737:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699245.721741:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699245.721744:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736000:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699245.721760:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699242/real 1787699245] req@ffff9d85b9d2ca80 x1874531134736128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699245.721773:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134736128/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699245.721780:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699245.721784:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699245.721787:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134736128:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699245.721794:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699242/real 1787699245] req@ffff9d85b9d2f480 x1874531134736256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699245.721806:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134736256/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699245.721814:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699245.721818:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699245.721821:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134736256:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699245.721827:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699242/real 1787699245] req@ffff9d85b9d2dc00 x1874531134736384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699245.721837:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134736384/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699245.721844:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699245.721847:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699245.722040:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134736384:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699246.291210:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699247.315145:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699247.763789:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699247.763803:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699247.763807:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699247.763820:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7149 00000100:00080000:0.0:1787699247.763834:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699247.763844:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699247.763936:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134736512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699247.763970:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699247.763974:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699247.763976:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699247.763980:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7149 00000100:00080000:0.0:1787699247.763984:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699247.763988:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699247.763999:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699247.764013:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134736640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699247.764023:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699247.764026:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699247.764028:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699247.764032:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7149 00000100:00080000:0.0:1787699247.764036:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699247.764039:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699247.764045:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699247.764055:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134736768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699247.764062:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699247.764066:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699247.764068:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699247.764071:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7149 00000100:00080000:0.0:1787699247.764075:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699247.764080:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699247.764084:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699247.764094:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134736896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699247.764102:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699247.764105:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699247.764107:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699247.764110:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7149 00000100:00080000:0.0:1787699247.764113:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699247.764116:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699247.764121:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699247.764131:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134737024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699247.764139:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699247.764142:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699247.764144:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699247.764148:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7149 00000100:00080000:0.0:1787699247.764151:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699247.764155:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699247.764160:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699247.764169:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134737152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699247.764182:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699247.764184:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699247.764186:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699247.764189:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7149 00000100:00080000:0.0:1787699247.764192:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699247.764196:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699247.764200:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699247.764210:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134737280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699247.764320:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134736512:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699247.764415:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699247.764476:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699247/real 1787699247] req@ffff9d85b9d09c00 x1874531134736512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699247.764496:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134736512/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699247.764508:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699247.764512:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699247.764516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134736512:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699247.764524:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736640:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699247.764547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699247.764559:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699247/real 1787699247] req@ffff9d85b9d08700 x1874531134736640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699247.764570:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134736640/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699247.764576:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699247.764579:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699247.764582:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736640:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699247.764588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736768:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699247.764607:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699247.764615:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699247/real 1787699247] req@ffff9d85b9d09500 x1874531134736768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699247.764625:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134736768/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699247.764631:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699247.764634:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699247.764637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736768:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699247.764642:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736896:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699247.764659:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699247.764668:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699247/real 1787699247] req@ffff9d85b9d09180 x1874531134736896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699247.764679:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134736896/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699247.764687:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699247.764690:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699247.764693:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134736896:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699247.764698:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737024:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699247.764729:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699247.764742:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699247/real 1787699247] req@ffff9d85b9d0a300 x1874531134737024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699247.764752:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134737024/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699247.764761:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699247.764764:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699247.764767:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737024:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699247.764773:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737152:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699247.764791:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699247.764800:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699247/real 1787699247] req@ffff9d85b9d09f80 x1874531134737152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699247.764811:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134737152/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699247.764825:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699247.764827:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699247.764831:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737152:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699247.764836:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737280:192.168.201.137@tcp:8:kworker.0 00000800:00000100:2.0:1787699247.765065:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:2.0:1787699247.765083:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699247/real 1787699247] req@ffff9d85b9d0b800 x1874531134737280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699247.765097:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134737280/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699247.765105:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699247.765108:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:2.0:1787699247.765111:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737280:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699248.339164:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699249.367960:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699250.387399:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699251.411100:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699252.435151:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699252.883058:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699252.883069:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699252.883074:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699252.883095:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7154 00000100:00080000:2.0:1787699252.883106:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699252.883113:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699252.883153:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5c00 x1874531134737408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699252.883184:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699252.883188:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699252.883190:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699252.883195:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7154 00000100:00080000:2.0:1787699252.883198:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699252.883202:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699252.883212:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699252.883224:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4000 x1874531134737536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699252.883234:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699252.883237:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699252.883239:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699252.883242:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7154 00000100:00080000:2.0:1787699252.883246:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699252.883249:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699252.883255:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699252.883266:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5500 x1874531134737664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699252.883275:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699252.883278:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699252.883280:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699252.883284:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7154 00000100:00080000:2.0:1787699252.883287:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699252.883290:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699252.883295:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699252.883304:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7800 x1874531134737792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699252.883312:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699252.883315:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699252.883317:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699252.883320:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7154 00000100:00080000:2.0:1787699252.883323:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699252.883327:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699252.883332:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699252.883341:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6d80 x1874531134737920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699252.883348:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699252.883351:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699252.883353:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699252.883356:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7154 00000100:00080000:2.0:1787699252.883359:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699252.883362:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699252.883366:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699252.883376:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7480 x1874531134738048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699252.883384:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699252.883387:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699252.883389:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699252.883392:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7154 00000100:00080000:2.0:1787699252.883395:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699252.883398:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699252.883402:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699252.883413:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6300 x1874531134738176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699252.883537:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134737408:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699252.883597:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699252.883644:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699252/real 1787699252] req@ffff9d85babd5c00 x1874531134737408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699252.883660:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5c00 x1874531134737408/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699252.883671:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699252.883675:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699252.883678:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134737408:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699252.883686:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134737536:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699252.883707:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699252.883719:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699252/real 1787699252] req@ffff9d85babd4000 x1874531134737536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699252.883731:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4000 x1874531134737536/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699252.883738:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699252.883741:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699252.883744:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134737536:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699252.883750:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134737664:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699252.883768:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699252.883779:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699252/real 1787699252] req@ffff9d85babd5500 x1874531134737664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699252.883790:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5500 x1874531134737664/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699252.883796:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699252.883799:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699252.883802:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134737664:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699252.883807:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134737792:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699252.883824:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699252.883833:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699252/real 1787699252] req@ffff9d85babd7800 x1874531134737792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699252.883843:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7800 x1874531134737792/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699252.883876:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699252.883879:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699252.883882:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134737792:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699252.883887:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737920:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699252.883906:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699252.883917:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699252/real 1787699252] req@ffff9d85babd6d80 x1874531134737920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699252.883926:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6d80 x1874531134737920/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699252.883933:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699252.883935:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699252.883938:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134737920:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699252.883943:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738048:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699252.883959:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699252.883968:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699252/real 1787699252] req@ffff9d85babd7480 x1874531134738048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699252.883977:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7480 x1874531134738048/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699252.883984:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699252.883986:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699252.883989:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738048:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699252.883994:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738176:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699252.884010:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699252.884018:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699252/real 1787699252] req@ffff9d85babd6300 x1874531134738176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699252.884028:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6300 x1874531134738176/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699252.884034:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699252.884037:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699252.884040:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738176:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699253.459289:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699254.483180:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699255.510649:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699256.532504:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699257.555140:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699258.002977:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699258.002986:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699258.002991:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699258.003002:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7159 00000100:00080000:3.0:1787699258.003012:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699258.003019:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699258.003055:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134738304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699258.003088:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699258.003092:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699258.003094:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699258.003098:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7159 00000100:00080000:3.0:1787699258.003102:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699258.003105:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699258.003114:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699258.003126:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134738432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699258.003136:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699258.003140:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699258.003141:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699258.003145:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7159 00000100:00080000:3.0:1787699258.003148:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699258.003152:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699258.003157:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699258.003165:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134738560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699258.003174:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699258.003177:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699258.003179:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699258.003182:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7159 00000100:00080000:3.0:1787699258.003185:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699258.003188:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699258.003192:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699258.003202:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134738688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699258.003209:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699258.003211:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699258.003213:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699258.003215:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7159 00000100:00080000:3.0:1787699258.003218:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699258.003221:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699258.003225:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699258.003233:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134738816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699258.003241:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699258.003244:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699258.003246:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699258.003249:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7159 00000100:00080000:3.0:1787699258.003252:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699258.003255:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699258.003259:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699258.003269:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134738944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699258.003278:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699258.003281:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699258.003282:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699258.003286:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7159 00000100:00080000:3.0:1787699258.003288:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699258.003291:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699258.003296:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699258.003305:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134739072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699258.003392:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134738304:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699258.003462:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699258.003502:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699258/real 1787699258] req@ffff9d85c0fb1f80 x1874531134738304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699258.003517:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134738304/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699258.003526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699258.003529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699258.003532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134738304:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699258.003539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134738432:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699258.003572:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699258.003583:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699258/real 1787699258] req@ffff9d85c0fb0700 x1874531134738432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699258.003595:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134738432/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699258.003603:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699258.003606:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699258.003609:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134738432:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699258.003615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134738560:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699258.003634:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699258.003645:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699258/real 1787699258] req@ffff9d85c0fb3100 x1874531134738560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699258.003655:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134738560/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699258.003662:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699258.003665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699258.003667:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134738560:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699258.003672:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134738688:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699258.003689:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699258.003699:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699258/real 1787699258] req@ffff9d85c0fb1880 x1874531134738688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699258.003710:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134738688/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699258.003717:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699258.003720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699258.003722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134738688:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699258.003728:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738816:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699258.003745:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699258.003755:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699258/real 1787699258] req@ffff9d85c0fb1500 x1874531134738816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699258.003764:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134738816/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699258.003771:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699258.003773:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699258.003776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738816:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699258.003781:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738944:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699258.003797:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699258.003806:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699258/real 1787699258] req@ffff9d85c0fb2300 x1874531134738944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699258.003816:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134738944/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699258.003822:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699258.003825:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699258.003828:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134738944:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699258.003833:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739072:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699258.003846:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699258.003882:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699258/real 1787699258] req@ffff9d85c0fb3480 x1874531134739072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699258.003892:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134739072/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699258.003899:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699258.003901:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699258.003904:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739072:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699258.579573:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699259.603088:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699260.627242:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699261.651060:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699262.677269:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699263.122991:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699263.123004:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699263.123008:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699263.123021:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7164 00000100:00080000:3.0:1787699263.123035:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699263.123043:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699263.123085:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134739200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699263.123117:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699263.123120:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699263.123122:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699263.123126:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7164 00000100:00080000:3.0:1787699263.123130:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699263.123133:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699263.123148:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699263.123160:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134739328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699263.123170:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699263.123172:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699263.123174:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699263.123178:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7164 00000100:00080000:3.0:1787699263.123181:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699263.123184:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699263.123188:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699263.123197:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134739456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699263.123205:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699263.123208:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699263.123210:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699263.123213:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7164 00000100:00080000:3.0:1787699263.123215:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699263.123218:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699263.123221:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699263.123230:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134739584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699263.123238:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699263.123241:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699263.123243:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699263.123246:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7164 00000100:00080000:3.0:1787699263.123248:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699263.123251:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699263.123255:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699263.123262:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134739712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699263.123270:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699263.123274:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699263.123275:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699263.123279:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7164 00000100:00080000:3.0:1787699263.123281:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699263.123284:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699263.123288:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699263.123296:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134739840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699263.123304:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699263.123308:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699263.123309:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699263.123313:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7164 00000100:00080000:3.0:1787699263.123315:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699263.123318:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699263.123321:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699263.123329:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134739968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699263.123509:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134739200:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699263.123570:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699263.123606:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699263/real 1787699263] req@ffff9d85c0fb1c00 x1874531134739200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699263.123622:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134739200/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699263.123632:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699263.123635:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699263.123639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134739200:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699263.123645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134739328:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699263.123664:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699263.123674:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699263/real 1787699263] req@ffff9d85c0fb0a80 x1874531134739328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699263.123685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134739328/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699263.123693:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699263.123696:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699263.123699:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134739328:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699263.123704:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134739456:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699263.123719:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699263.123727:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699263/real 1787699263] req@ffff9d85c0fb3b80 x1874531134739456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699263.123738:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134739456/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699263.123745:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699263.123748:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699263.123751:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134739456:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699263.123756:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134739584:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699263.123770:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699263.123792:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699263/real 1787699263] req@ffff9d85c0fb0380 x1874531134739584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699263.123803:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134739584/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699263.123811:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699263.123814:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699263.123817:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134739584:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699263.123822:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739712:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699263.123839:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699263.123847:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699263/real 1787699263] req@ffff9d85c0fb2d80 x1874531134739712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699263.123893:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134739712/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699263.123902:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699263.124096:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699263.124103:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739712:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699263.124110:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739840:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699263.124131:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699263.124141:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699263/real 1787699263] req@ffff9d85c0fb3800 x1874531134739840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699263.124152:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134739840/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699263.124160:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699263.124163:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699263.124166:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739840:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699263.124171:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739968:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699263.124185:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699263.124194:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699263/real 1787699263] req@ffff9d85c0fb0000 x1874531134739968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699263.124203:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134739968/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699263.124211:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699263.124214:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699263.124217:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134739968:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699263.699194:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699264.732921:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699265.747070:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699266.771025:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699267.795095:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699268.246064:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699268.246076:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699268.246081:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699268.246091:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7170 00000100:00080000:2.0:1787699268.246102:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699268.246109:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699268.246148:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4380 x1874531134740096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699268.246186:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699268.246190:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699268.246192:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699268.246196:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7170 00000100:00080000:2.0:1787699268.246200:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699268.246203:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699268.246212:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699268.246225:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6680 x1874531134740224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699268.246234:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699268.246237:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699268.246239:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699268.246242:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7170 00000100:00080000:2.0:1787699268.246246:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699268.246249:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699268.246254:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699268.246264:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd6a00 x1874531134740352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699268.246273:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699268.246276:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699268.246278:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699268.246281:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7170 00000100:00080000:2.0:1787699268.246285:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699268.246288:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699268.246292:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699268.246303:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134740480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699268.246312:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699268.246315:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699268.246316:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699268.246319:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7170 00000100:00080000:2.0:1787699268.246321:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699268.246324:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699268.246328:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699268.246337:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134740608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699268.246345:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699268.246347:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699268.246349:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699268.246351:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7170 00000100:00080000:2.0:1787699268.246354:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699268.246358:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699268.246362:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699268.246403:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134740736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699268.246421:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699268.246424:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699268.246425:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699268.246428:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7170 00000100:00080000:2.0:1787699268.246431:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699268.246435:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699268.246439:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699268.246449:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134740864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699268.246543:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134740096:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699268.246608:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699268.246647:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699268/real 1787699268] req@ffff9d85babd4380 x1874531134740096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699268.246664:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4380 x1874531134740096/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699268.246674:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699268.246677:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699268.246681:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134740096:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699268.246688:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134740224:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699268.246709:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699268.246722:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699268/real 1787699268] req@ffff9d85babd6680 x1874531134740224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699268.246732:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6680 x1874531134740224/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699268.246740:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699268.246743:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699268.246746:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134740224:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699268.246752:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134740352:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699268.246771:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699268.246781:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699268/real 1787699268] req@ffff9d85babd6a00 x1874531134740352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699268.246791:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd6a00 x1874531134740352/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699268.246797:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699268.246800:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699268.246803:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134740352:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699268.246808:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134740480:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699268.246826:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699268.246836:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699268/real 1787699268] req@ffff9d85babd4e00 x1874531134740480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699268.246846:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134740480/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699268.246890:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699268.246893:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699268.246896:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134740480:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699268.246901:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134740608:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699268.246922:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699268.246933:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699268/real 1787699268] req@ffff9d85babd7b80 x1874531134740608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699268.246943:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134740608/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699268.246949:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699268.246952:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699268.246955:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134740608:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699268.246961:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134740736:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699268.246979:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699268.246989:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699268/real 1787699268] req@ffff9d85babd5f80 x1874531134740736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699268.247000:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134740736/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699268.247007:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699268.247009:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699268.247013:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134740736:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699268.247018:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134740864:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699268.247036:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699268.247046:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699268/real 1787699268] req@ffff9d85babd4700 x1874531134740864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699268.247057:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134740864/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699268.247065:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699268.247067:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699268.247070:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134740864:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699268.822768:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699269.847905:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699270.867112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699271.893018:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699272.915051:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699273.362976:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699273.362987:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699273.362990:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699273.363001:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7175 00000100:00080000:2.0:1787699273.363012:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699273.363020:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:2.0:1787699273.363059:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5180 x1874531134740992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699273.363097:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699273.363101:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699273.363103:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699273.363108:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7175 00000100:00080000:2.0:1787699273.363112:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699273.363116:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699273.363125:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699273.363141:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7100 x1874531134741120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699273.363151:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699273.363154:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699273.363155:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699273.363159:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7175 00000100:00080000:2.0:1787699273.363163:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699273.363166:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699273.363171:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699273.363180:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4a80 x1874531134741248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699273.363188:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699273.363191:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699273.363193:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699273.363195:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7175 00000100:00080000:2.0:1787699273.363199:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699273.363202:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699273.363205:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699273.363213:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4700 x1874531134741376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699273.363221:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699273.363224:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699273.363225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699273.363228:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7175 00000100:00080000:2.0:1787699273.363231:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699273.363235:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699273.363238:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699273.363247:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd5f80 x1874531134741504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699273.363255:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699273.363259:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699273.363260:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699273.363263:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7175 00000100:00080000:2.0:1787699273.363266:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699273.363270:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699273.363274:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699273.363293:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd7b80 x1874531134741632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699273.363308:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699273.363311:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699273.363313:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699273.363316:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7175 00000100:00080000:2.0:1787699273.363320:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699273.363323:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:2.0:1787699273.363327:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699273.363338:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85babd4e00 x1874531134741760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699273.363432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134740992:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699273.363496:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699273.363540:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699273/real 1787699273] req@ffff9d85babd5180 x1874531134740992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699273.363558:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5180 x1874531134740992/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699273.363569:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699273.363572:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699273.363576:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134740992:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699273.363583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134741120:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699273.363605:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699273.363618:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699273/real 1787699273] req@ffff9d85babd7100 x1874531134741120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699273.363630:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7100 x1874531134741120/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699273.363638:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699273.363642:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699273.363645:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134741120:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699273.363650:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134741248:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699273.363669:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699273.363680:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699273/real 1787699273] req@ffff9d85babd4a80 x1874531134741248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699273.363691:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4a80 x1874531134741248/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699273.363699:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699273.363701:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699273.363705:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134741248:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699273.363711:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134741376:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699273.363730:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699273.363741:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699273/real 1787699273] req@ffff9d85babd4700 x1874531134741376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699273.363753:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4700 x1874531134741376/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699273.363761:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699273.363764:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699273.363767:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134741376:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699273.363771:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134741504:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699273.363874:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699273.363889:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699273/real 1787699273] req@ffff9d85babd5f80 x1874531134741504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699273.363902:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd5f80 x1874531134741504/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699273.363909:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699273.363913:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699273.363916:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134741504:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699273.363921:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134741632:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699273.363943:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699273.363954:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699273/real 1787699273] req@ffff9d85babd7b80 x1874531134741632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699273.363965:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd7b80 x1874531134741632/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699273.363973:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699273.363976:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699273.363980:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134741632:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699273.363985:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134741760:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699273.364004:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699273.364014:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699273/real 1787699273] req@ffff9d85babd4e00 x1874531134741760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699273.364031:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85babd4e00 x1874531134741760/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699273.364039:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699273.364042:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699273.364044:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85babd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134741760:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699273.939109:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699274.963422:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699275.987314:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699277.011337:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699278.035498:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699278.484937:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699278.484948:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699278.484953:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699278.484965:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7180 00000100:00080000:3.0:1787699278.484980:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699278.484988:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699278.485038:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134741888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699278.485069:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699278.485072:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699278.485074:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699278.485077:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7180 00000100:00080000:3.0:1787699278.485079:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699278.485082:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699278.485092:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699278.485100:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134742016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699278.485106:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699278.485109:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699278.485110:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699278.485112:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7180 00000100:00080000:3.0:1787699278.485113:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699278.485115:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699278.485118:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699278.485123:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134742144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699278.485132:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699278.485135:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699278.485136:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699278.485138:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7180 00000100:00080000:3.0:1787699278.485140:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699278.485142:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699278.485144:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699278.485152:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134742272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699278.485157:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699278.485159:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699278.485159:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699278.485161:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7180 00000100:00080000:3.0:1787699278.485163:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699278.485165:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699278.485168:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699278.485177:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134742400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699278.485186:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699278.485189:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699278.485191:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699278.485226:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7180 00000100:00080000:3.0:1787699278.485230:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699278.485235:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699278.485239:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699278.485247:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134742528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699278.485254:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699278.485257:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699278.485258:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699278.485261:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7180 00000100:00080000:3.0:1787699278.485263:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699278.485266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699278.485268:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699278.485272:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134742656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699278.485410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134741888:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699278.485499:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699278.485573:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699278/real 1787699278] req@ffff9d85c0fb0e00 x1874531134741888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699278.485588:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134741888/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699278.485596:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699278.485599:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699278.485602:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134741888:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699278.485608:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742016:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699278.485626:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699278.485634:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699278/real 1787699278] req@ffff9d85c0fb2680 x1874531134742016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699278.485641:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134742016/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699278.485651:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699278.485653:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699278.485655:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742016:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699278.485658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742144:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699278.485669:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699278.485675:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699278/real 1787699278] req@ffff9d85c0fb2a00 x1874531134742144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699278.485682:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134742144/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699278.485687:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699278.485689:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699278.485690:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742144:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699278.485693:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742272:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699278.485704:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699278.485708:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699278/real 1787699278] req@ffff9d85c0fb0000 x1874531134742272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699278.485714:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134742272/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699278.485730:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699278.485733:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699278.485737:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742272:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699278.485742:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134742400:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699278.485762:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699278.485773:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699278/real 1787699278] req@ffff9d85c0fb3800 x1874531134742400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699278.485782:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134742400/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699278.485789:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699278.485792:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699278.485794:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134742400:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699278.485799:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134742528:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699278.485816:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699278.485826:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699278/real 1787699278] req@ffff9d85c0fb2d80 x1874531134742528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699278.485836:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134742528/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699278.485842:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699278.485845:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699278.485848:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134742528:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699278.485895:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134742656:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699278.485915:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699278.485930:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699278/real 1787699278] req@ffff9d85c0fb0380 x1874531134742656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699278.485942:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134742656/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699278.485950:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699278.485953:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699278.485956:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134742656:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699279.059375:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699280.083691:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699281.107657:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699282.131117:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699283.158223:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699283.602960:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699283.602970:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699283.602975:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699283.602983:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7185 00000100:00080000:3.0:1787699283.602994:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699283.603000:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699283.603032:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134742784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699283.603070:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699283.603074:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699283.603075:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699283.603079:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7185 00000100:00080000:3.0:1787699283.603082:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699283.603086:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699283.603094:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699283.603104:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134742912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699283.603129:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699283.603132:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699283.603134:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699283.603137:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7185 00000100:00080000:3.0:1787699283.603140:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699283.603143:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699283.603148:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699283.603157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134743040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699283.603165:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699283.603168:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699283.603170:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699283.603174:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7185 00000100:00080000:3.0:1787699283.603177:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699283.603180:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699283.603183:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699283.603191:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134743168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699283.603199:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699283.603202:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699283.603204:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699283.603207:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7185 00000100:00080000:3.0:1787699283.603209:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699283.603213:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699283.603216:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699283.603223:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134743296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699283.603231:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699283.603234:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699283.603235:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699283.603238:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7185 00000100:00080000:3.0:1787699283.603241:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699283.603244:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699283.603248:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699283.603255:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134743424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699283.603269:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699283.603272:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699283.603274:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699283.603277:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7185 00000100:00080000:3.0:1787699283.603280:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699283.603283:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699283.603287:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699283.603296:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134743552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699283.603390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134742784:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699283.603453:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699283.603488:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699283/real 1787699283] req@ffff9d85c0fb3b80 x1874531134742784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.603504:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134742784/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.603513:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.603516:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699283.603520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134742784:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699283.603527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742912:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699283.603547:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699283.603558:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699283/real 1787699283] req@ffff9d85c0fb0a80 x1874531134742912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.603568:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134742912/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.603582:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.603585:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699283.603588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134742912:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699283.603594:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743040:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699283.603610:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699283.603618:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699283/real 1787699283] req@ffff9d85c0fb1c00 x1874531134743040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.603628:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134743040/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.603636:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.603639:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699283.603642:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743040:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699283.603648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743168:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699283.603661:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699283.603670:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699283/real 1787699283] req@ffff9d85c0fb3480 x1874531134743168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.603681:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134743168/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.603688:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.603691:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699283.603694:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743168:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699283.603699:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134743296:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699283.603713:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699283.603722:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699283/real 1787699283] req@ffff9d85c0fb2300 x1874531134743296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.603732:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134743296/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.603739:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.603742:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699283.603745:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134743296:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699283.603750:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134743424:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699283.603762:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699283.603770:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699283/real 1787699283] req@ffff9d85c0fb1500 x1874531134743424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.603780:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134743424/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.603787:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.603789:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699283.603793:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134743424:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699283.603798:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134743552:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699283.603810:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699283.603819:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699283/real 1787699283] req@ffff9d85c0fb1880 x1874531134743552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.603829:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134743552/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.603836:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.603839:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699283.603842:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134743552:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699284.179066:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699285.202993:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699286.227152:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699287.255134:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699288.275077:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699288.722953:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699288.722960:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699288.722964:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699288.722971:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7190 00000100:00080000:3.0:1787699288.722981:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699288.722987:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699288.723019:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3100 x1874531134743680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699288.723073:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699288.723077:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699288.723078:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699288.723082:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7190 00000100:00080000:3.0:1787699288.723085:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699288.723087:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699288.723094:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699288.723103:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0700 x1874531134743808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699288.723110:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699288.723112:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699288.723113:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699288.723116:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7190 00000100:00080000:3.0:1787699288.723117:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699288.723120:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699288.723123:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699288.723130:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1f80 x1874531134743936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699288.723132:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134743680:192.168.201.137@tcp:250:kworker.0 00000100:00080000:3.0:1787699288.723138:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699288.723142:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699288.723143:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699288.723145:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7190 00000100:00080000:3.0:1787699288.723146:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699288.723148:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699288.723150:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699288.723157:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1880 x1874531134744064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699288.723163:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699288.723165:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699288.723166:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699288.723168:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7190 00000100:00080000:3.0:1787699288.723169:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699288.723173:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699288.723177:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699288.723197:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1500 x1874531134744192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699288.723205:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787699288.723211:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699288.723212:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699288.723215:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699288.723220:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7190 00000100:00080000:3.0:1787699288.723225:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699288.723229:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699288.723234:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699288.723245:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2300 x1874531134744320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699288.723253:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699288.723255:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699288.723256:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699288.723260:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7190 00000100:00080000:3.0:1787699288.723263:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699288.723266:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699288.723270:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699288.723279:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3480 x1874531134744448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699288.723290:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699288/real 1787699288] req@ffff9d85c0fb3100 x1874531134743680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699288.723311:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3100 x1874531134743680/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699288.723331:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699288.723335:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699288.723339:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134743680:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699288.723346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743808:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699288.723375:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699288.723387:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699288/real 1787699288] req@ffff9d85c0fb0700 x1874531134743808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699288.723399:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0700 x1874531134743808/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699288.723413:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699288.723416:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699288.723419:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743808:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699288.723444:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743936:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699288.723472:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699288.723484:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699288/real 1787699288] req@ffff9d85c0fb1f80 x1874531134743936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699288.723494:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1f80 x1874531134743936/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699288.723501:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699288.723504:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699288.723507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134743936:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699288.723512:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744064:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699288.723528:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699288.723548:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699288/real 1787699288] req@ffff9d85c0fb1880 x1874531134744064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699288.723559:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1880 x1874531134744064/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699288.723566:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699288.723569:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699288.723572:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744064:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699288.723578:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134744192:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699288.723596:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699288.723606:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699288/real 1787699288] req@ffff9d85c0fb1500 x1874531134744192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699288.723625:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1500 x1874531134744192/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699288.723632:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699288.723635:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699288.723638:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134744192:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699288.723643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134744320:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699288.723662:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699288.723672:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699288/real 1787699288] req@ffff9d85c0fb2300 x1874531134744320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699288.723683:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2300 x1874531134744320/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699288.723701:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699288.723705:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699288.723708:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134744320:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699288.723713:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134744448:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699288.723732:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699288.723752:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699288/real 1787699288] req@ffff9d85c0fb3480 x1874531134744448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699288.723763:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3480 x1874531134744448/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699288.723771:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699288.723774:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699288.723778:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134744448:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699289.300036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699290.323016:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699291.347031:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699292.372689:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699293.395874:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699293.842920:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699293.842931:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699293.842946:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699293.842959:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7195 00000100:00080000:3.0:1787699293.842970:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699293.842979:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699293.843028:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb1c00 x1874531134744576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699293.843306:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134744576:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699293.843379:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699293.843416:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699293.843421:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699293.843423:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699293.843429:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7195 00000100:00080000:3.0:1787699293.843434:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699293.843438:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699293.843448:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699293.843463:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0a80 x1874531134744704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699293.843796:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699293/real 1787699293] req@ffff9d85c0fb1c00 x1874531134744576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699293.843815:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb1c00 x1874531134744576/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699293.843826:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699293.843830:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699293.843835:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134744576:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699293.843898:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744704:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699293.843927:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699293.843977:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699293/real 1787699293] req@ffff9d85c0fb0a80 x1874531134744704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699293.843990:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0a80 x1874531134744704/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699293.844005:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699293.844007:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699293.844010:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744704:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699293.847604:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699293.847616:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699293.847620:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699293.847631:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7195 00000100:00080000:3.0:1787699293.847641:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699293.847660:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699293.847673:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699293.847712:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3b80 x1874531134744832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699293.847726:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699293.847730:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699293.847731:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699293.847735:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7195 00000100:00080000:3.0:1787699293.847739:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699293.847743:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699293.847747:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699293.847759:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0380 x1874531134744960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699293.847769:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699293.847772:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699293.847774:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699293.847777:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7195 00000100:00080000:3.0:1787699293.847781:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699293.847784:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699293.847789:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699293.847798:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2d80 x1874531134745088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699293.847808:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699293.847811:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699293.847812:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699293.847816:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7195 00000100:00080000:3.0:1787699293.847819:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699293.847822:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699293.847827:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699293.847837:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134745216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699293.847846:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699293.847849:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699293.847866:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699293.847870:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7195 00000100:00080000:3.0:1787699293.847873:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699293.847877:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699293.847881:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699293.847891:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134745344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699293.848050:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744832:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699293.848248:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699293.848275:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699293/real 1787699293] req@ffff9d85c0fb3b80 x1874531134744832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699293.848293:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3b80 x1874531134744832/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699293.848302:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699293.848307:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699293.848310:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744832:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699293.848318:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744960:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699293.848338:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699293.848348:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699293/real 1787699293] req@ffff9d85c0fb0380 x1874531134744960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699293.848358:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0380 x1874531134744960/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699293.848366:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699293.848369:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699293.848372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134744960:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699293.848378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745088:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699293.848396:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699293.848407:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699293/real 1787699293] req@ffff9d85c0fb2d80 x1874531134745088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699293.848416:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2d80 x1874531134745088/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699293.848422:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699293.848425:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699293.848429:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745088:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699293.848434:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745216:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699293.848452:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699293.848462:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699293/real 1787699293] req@ffff9d85c0fb3800 x1874531134745216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699293.848472:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134745216/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699293.848480:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699293.848483:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699293.848486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745216:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699293.848491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745344:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699293.848507:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699293.848517:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699293/real 1787699293] req@ffff9d85c0fb0000 x1874531134745344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699293.848528:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134745344/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699293.848535:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699293.848538:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699293.848540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745344:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699294.420011:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699295.443707:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699296.467165:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699297.491634:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699298.515071:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699298.963922:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699298.963930:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699298.963933:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699298.963943:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7200 00000100:00080000:1.0:1787699298.963953:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699298.963960:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699298.963997:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134745472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699298.964903:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134745472:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699298.964962:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699298.966218:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699298.966225:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699298.966229:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699298.966239:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7200 00000100:00080000:1.0:1787699298.966248:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699298.966255:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699298.966266:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699298.966293:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e680 x1874531134745600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699298.966343:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699298.966347:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699298.966348:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699298.966352:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7200 00000100:00080000:1.0:1787699298.966355:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699298.966359:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699298.966363:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699298.966375:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ed80 x1874531134745728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699298.966383:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699298.966386:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699298.966387:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699298.966390:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7200 00000100:00080000:1.0:1787699298.966393:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699298.966395:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699298.966399:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699298.966408:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c700 x1874531134745856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699298.966409:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699298/real 1787699298] req@ffff9d85b9d2d500 x1874531134745472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699298.966417:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699298.966420:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699298.966421:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699298.966424:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7200 00000100:00100000:0.0:1787699298.966427:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134745472/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699298.966427:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699298.966431:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699298.966435:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699298.966437:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699298.966441:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699298.966442:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c380 x1874531134745984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699298.966444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134745472:192.168.201.137@tcp:250:kworker.0 00000100:00080000:1.0:1787699298.966450:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699298.966453:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699298.966454:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699298.966457:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7200 00000100:00080000:1.0:1787699298.966459:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699298.966462:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699298.966465:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699298.966468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134745600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699298.966474:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2e300 x1874531134746112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699298.966482:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699298.966485:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699298.966486:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699298.966489:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7200 00000100:00080000:1.0:1787699298.966492:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699298.966495:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:1.0:1787699298.966495:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699298.966499:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699298.966509:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699298/real 1787699298] req@ffff9d85b9d2e680 x1874531134745600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699298.966510:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2d500 x1874531134746240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699298.966521:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e680 x1874531134745600/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699298.966528:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699298.966530:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699298.966534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134745600:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699298.966539:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134745728:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699298.966557:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699298.966567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699298/real 1787699298] req@ffff9d85b9d2ed80 x1874531134745728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699298.966578:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ed80 x1874531134745728/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699298.966585:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699298.966587:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699298.966589:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134745728:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699298.966593:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134745856:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699298.966606:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699298.966612:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699298/real 1787699298] req@ffff9d85b9d2c700 x1874531134745856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699298.966618:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c700 x1874531134745856/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699298.966623:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699298.966625:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699298.966627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134745856:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699298.966629:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745984:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699298.966638:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699298.966643:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699298/real 1787699298] req@ffff9d85b9d2c380 x1874531134745984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699298.966649:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c380 x1874531134745984/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699298.966653:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699298.966655:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699298.966656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134745984:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699298.966665:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134746112:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699298.966674:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699298.966679:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699298/real 1787699298] req@ffff9d85b9d2e300 x1874531134746112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699298.966685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2e300 x1874531134746112/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699298.966689:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699298.966691:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699298.966693:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134746112:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699298.966696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134746240:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699298.966704:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699298.966708:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699298/real 1787699298] req@ffff9d85b9d2d500 x1874531134746240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699298.966714:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2d500 x1874531134746240/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699298.966718:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699298.966720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699298.966721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134746240:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699299.539572:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699300.564280:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699301.588972:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699302.614293:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699303.635209:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699304.082973:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699304.082983:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699304.082987:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699304.082997:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7205 00000100:00080000:1.0:1787699304.083007:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699304.083013:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:1.0:1787699304.083047:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2dc00 x1874531134746368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699304.083077:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699304.083081:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699304.083083:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699304.083086:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7205 00000100:00080000:1.0:1787699304.083089:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699304.083093:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699304.083102:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699304.083112:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f480 x1874531134746496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699304.083121:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699304.083124:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699304.083126:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699304.083129:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7205 00000100:00080000:1.0:1787699304.083132:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699304.083135:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699304.083139:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699304.083149:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ca80 x1874531134746624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699304.083157:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699304.083160:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699304.083161:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699304.083164:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7205 00000100:00080000:1.0:1787699304.083166:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699304.083169:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699304.083173:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699304.083181:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2f800 x1874531134746752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699304.083189:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699304.083192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699304.083193:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699304.083196:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7205 00000100:00080000:1.0:1787699304.083198:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699304.083202:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699304.083206:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699304.083213:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2ce00 x1874531134746880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699304.083220:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699304.083223:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699304.083225:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699304.083229:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7205 00000100:00080000:1.0:1787699304.083231:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699304.083234:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699304.083238:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699304.083247:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2c000 x1874531134747008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699304.083254:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699304.083257:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699304.083259:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699304.083262:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7205 00000100:00080000:1.0:1787699304.083265:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699304.083268:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:1.0:1787699304.083272:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699304.083283:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d2fb80 x1874531134747136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699304.083378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134746368:192.168.201.137@tcp:250:kworker.0 00000800:00000100:0.0:1787699304.083436:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699304.083471:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699304/real 1787699304] req@ffff9d85b9d2dc00 x1874531134746368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699304.083487:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2dc00 x1874531134746368/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699304.083496:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699304.083500:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699304.083504:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134746368:192.168.201.137@tcp:250:kworker.0 00000100:00100000:0.0:1787699304.083511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134746496:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699304.083539:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699304.083550:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699304/real 1787699304] req@ffff9d85b9d2f480 x1874531134746496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699304.083561:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f480 x1874531134746496/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699304.083568:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699304.083571:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699304.083574:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134746496:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699304.083579:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134746624:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699304.083594:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699304.083602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699304/real 1787699304] req@ffff9d85b9d2ca80 x1874531134746624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699304.083613:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ca80 x1874531134746624/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699304.083620:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699304.083623:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699304.083627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134746624:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699304.083631:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134746752:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699304.083644:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699304.083651:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699304/real 1787699304] req@ffff9d85b9d2f800 x1874531134746752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699304.083661:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2f800 x1874531134746752/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699304.083669:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699304.083672:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699304.083676:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134746752:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699304.083680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134746880:192.168.201.137@tcp:38:kworker.0 00000800:00000100:0.0:1787699304.083693:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699304.083701:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699304/real 1787699304] req@ffff9d85b9d2ce00 x1874531134746880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699304.083710:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2ce00 x1874531134746880/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699304.083718:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699304.083720:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699304.083723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134746880:192.168.201.137@tcp:38:kworker.0 00000100:00100000:0.0:1787699304.083728:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747008:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699304.083741:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699304.083749:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699304/real 1787699304] req@ffff9d85b9d2c000 x1874531134747008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699304.083772:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2c000 x1874531134747008/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699304.083780:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699304.083783:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699304.083787:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747008:192.168.201.137@tcp:8:kworker.0 00000100:00100000:0.0:1787699304.083792:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747136:192.168.201.137@tcp:8:kworker.0 00000800:00000100:0.0:1787699304.083808:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:0.0:1787699304.083817:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699304/real 1787699304] req@ffff9d85b9d2fb80 x1874531134747136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699304.083827:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d2fb80 x1874531134747136/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699304.083834:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699304.083837:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:0.0:1787699304.083840:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747136:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699304.659122:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699305.683691:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699306.707875:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699307.733338:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699308.756743:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699309.203028:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699309.203039:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699309.203043:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699309.203054:0:89493:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7210 00000100:00080000:0.0:1787699309.203065:0:89493:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699309.203072:0:89493:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:0.0:1787699309.203114:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0b800 x1874531134747264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699309.203143:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699309.203147:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699309.203149:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699309.203153:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7210 00000100:00080000:0.0:1787699309.203156:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699309.203160:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699309.203169:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699309.203182:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09f80 x1874531134747392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699309.203190:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699309.203193:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699309.203195:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699309.203197:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7210 00000100:00080000:0.0:1787699309.203201:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699309.203204:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699309.203208:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699309.203219:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d0a300 x1874531134747520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699309.203228:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699309.203231:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699309.203232:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699309.203236:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7210 00000100:00080000:0.0:1787699309.203238:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699309.203242:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699309.203246:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699309.203256:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09180 x1874531134747648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699309.203265:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699309.203268:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699309.203270:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699309.203273:0:89493:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7210 00000100:00080000:0.0:1787699309.203276:0:89493:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699309.203279:0:89493:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699309.203283:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699309.203292:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09500 x1874531134747776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699309.203300:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699309.203303:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699309.203304:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699309.203308:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7210 00000100:00080000:0.0:1787699309.203311:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699309.203314:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699309.203318:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699309.203328:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d08700 x1874531134747904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699309.203335:0:89493:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699309.203337:0:89493:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699309.203338:0:89493:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699309.203341:0:89493:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7210 00000100:00080000:0.0:1787699309.203344:0:89493:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699309.203347:0:89493:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:0.0:1787699309.203351:0:89493:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699309.203360:0:89493:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85b9d09c00 x1874531134748032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699309.203436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134747264:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699309.203533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134747392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699309.203561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134747520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699309.203582:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134747648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699309.203598:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699309.203617:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699309.203634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748032:192.168.201.137@tcp:8:kworker.0 00010000:00010000:2.0:1787699309.779034:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699310.803065:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787699311.831108:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787699312.275184:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.137:988 00000400:00000100:3.0:1787699312.275226:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.137@tcp at host 192.168.201.137 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787699312.275245:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699312.275285:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699312.275295:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699312.275303:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699312.275310:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699312.275317:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000800:00000100:3.0:1787699312.275325:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.137@tcp 00000100:00100000:1.0:1787699312.275455:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699309/real 1787699312] req@ffff9d85b9d0b800 x1874531134747264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699312.275480:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0b800 x1874531134747264/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699312.275491:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699312.275495:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699312.275499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134747264:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699312.275507:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699309/real 1787699312] req@ffff9d85b9d09f80 x1874531134747392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699312.275519:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09f80 x1874531134747392/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699312.275526:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699312.275529:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699312.275532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134747392:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699312.275538:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699309/real 1787699312] req@ffff9d85b9d0a300 x1874531134747520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699312.275548:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d0a300 x1874531134747520/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699312.275560:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699312.275563:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699312.275566:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d0a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134747520:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699312.275572:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699309/real 1787699312] req@ffff9d85b9d09180 x1874531134747648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699312.275582:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09180 x1874531134747648/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699312.275590:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699312.275593:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699312.275596:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134747648:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699312.275602:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699309/real 1787699312] req@ffff9d85b9d09500 x1874531134747776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699312.275612:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09500 x1874531134747776/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699312.275618:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699312.275633:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699312.275636:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747776:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699312.275643:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699309/real 1787699312] req@ffff9d85b9d08700 x1874531134747904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699312.275654:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d08700 x1874531134747904/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699312.275662:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699312.275665:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699312.275668:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d08700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134747904:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699312.275674:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699309/real 1787699312] req@ffff9d85b9d09c00 x1874531134748032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699312.275685:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85b9d09c00 x1874531134748032/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699312.275693:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699312.275696:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699312.275699:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85b9d09c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748032:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699312.851947:0:44688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699313.875309:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699314.329172:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73dc9a2d-769a-4716-a6c9-49ac63f28c0c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699314.329192:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699314.329198:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699314.329218:0:100989:0:(import.c:510:import_select_connection()) MGC192.168.201.137@tcp: connect to NID 192.168.201.137@tcp last attempt 7216 00000100:00080000:3.0:1787699314.329235:0:100989:0:(import.c:589:import_select_connection()) MGC192.168.201.137@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699314.329256:0:100989:0:(import.c:624:import_select_connection()) MGC192.168.201.137@tcp: import ffff9d858a6e6800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000100:00100000:3.0:1787699314.329522:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134748160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699314.329644:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134748160:192.168.201.137@tcp:250:kworker.0 00000800:00000100:1.0:1787699314.329742:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00080000:3.0:1787699314.329786:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699314.329791:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699314.329793:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699314.329798:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7216 00000100:00080000:3.0:1787699314.329804:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699314.329809:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d8590d43000: import ffff9d858c487000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699314.329822:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699314.329838:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134748288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699314.338460:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699314/real 1787699314] req@ffff9d85c0fb2a00 x1874531134748160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699314.338502:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134748160/t0(0) o250->MGC192.168.201.137@tcp@192.168.201.137@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699314.338515:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858a6e6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699314.338520:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699314.338524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73dc9a2d-769a-4716-a6c9-49ac63f28c0c:2359:1874531134748160:192.168.201.137@tcp:250:kworker.0 00000100:00100000:1.0:1787699314.338566:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134748288:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699314.339215:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699314.339240:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699314/real 1787699314] req@ffff9d85c0fb2680 x1874531134748288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699314.339959:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134748288/t0(0) o38->lustre-MDT0000-mdc-ffff9d8590d43000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699314.339971:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c487000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699314.339976:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699314.339981:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134748288:192.168.201.137@tcp:38:kworker.0 00000100:00080000:3.0:1787699314.340084:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699314.340098:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699314.340102:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699314.340109:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7216 00000100:00080000:3.0:1787699314.340119:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699314.340128:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d8590d43000: import ffff9d858c485000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699314.340142:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699314.340179:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0e00 x1874531134748416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699314.340194:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) 68e474e8-42cb-4207-86a3-041065654492->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699314.340198:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699314.340200:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699314.340203:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: connect to NID 192.168.201.137@tcp last attempt 7216 00000100:00080000:3.0:1787699314.340208:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699314.340212:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d8590d43000: import ffff9d858c484800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699314.340216:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699314.340229:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2680 x1874531134748544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699314.340239:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699314.340441:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699314.340444:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699314.340449:0:100989:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7216 00000100:00080000:3.0:1787699314.340453:0:100989:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699314.340458:0:100989:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9d858c4eb000: import ffff9d858c4ef800 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699314.340465:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699314.340478:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb2a00 x1874531134748672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699314.340488:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699314.340492:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699314.340494:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699314.340498:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7216 00000100:00080000:3.0:1787699314.340501:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699314.340505:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9d858c4eb000: import ffff9d858c4ec000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699314.340510:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699314.340520:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb0000 x1874531134748800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699314.340535:0:100989:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3d8e19c-573a-445a-9e98-5fe67f1b3715->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699314.340539:0:100989:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699314.340541:0:100989:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699314.340544:0:100989:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: connect to NID 192.168.201.137@tcp last attempt 7216 00000100:00080000:3.0:1787699314.340548:0:100989:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699314.340551:0:100989:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9d858c4eb000: import ffff9d858c4ed000 using connection 192.168.201.137@tcp/192.168.201.137@tcp 00000008:00100000:3.0:1787699314.340556:0:100989:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699314.340565:0:100989:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d85c0fb3800 x1874531134748928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699314.341157:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134748416:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699314.341226:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699314.341763:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699314/real 1787699314] req@ffff9d85c0fb0e00 x1874531134748416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699314.341785:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0e00 x1874531134748416/t0(0) o8->lustre-OST0000-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699314.341796:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c485000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699314.341802:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699314.341806:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134748416:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699314.341814:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134748544:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699314.341841:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699314.341879:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699314/real 1787699314] req@ffff9d85c0fb2680 x1874531134748544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699314.341898:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2680 x1874531134748544/t0(0) o8->lustre-OST0001-osc-ffff9d8590d43000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699314.341907:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c484800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699314.341910:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699314.341913:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:68e474e8-42cb-4207-86a3-041065654492:2359:1874531134748544:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699314.341920:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748672:192.168.201.137@tcp:38:kworker.0 00000800:00000100:1.0:1787699314.341942:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699314.341954:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699314/real 1787699314] req@ffff9d85c0fb2a00 x1874531134748672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699314.341965:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb2a00 x1874531134748672/t0(0) o38->lustre-MDT0000-mdc-ffff9d858c4eb000@192.168.201.137@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699314.341973:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ef800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699314.341976:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699314.341979:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748672:192.168.201.137@tcp:38:kworker.0 00000100:00100000:1.0:1787699314.341985:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748800:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699314.342004:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699314.342015:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699314/real 1787699314] req@ffff9d85c0fb0000 x1874531134748800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699314.342025:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb0000 x1874531134748800/t0(0) o8->lustre-OST0000-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699314.342039:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ec000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699314.342043:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699314.342046:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748800:192.168.201.137@tcp:8:kworker.0 00000100:00100000:1.0:1787699314.342053:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748928:192.168.201.137@tcp:8:kworker.0 00000800:00000100:1.0:1787699314.342072:0:2359:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.137@tcp 00000100:00100000:1.0:1787699314.342083:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699314/real 1787699314] req@ffff9d85c0fb3800 x1874531134748928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699314.342094:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d85c0fb3800 x1874531134748928/t0(0) o8->lustre-OST0001-osc-ffff9d858c4eb000@192.168.201.137@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699314.342102:0:2359:0:(import.c:65:import_set_state_nolock()) ffff9d858c4ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699314.342105:0:2359:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.137@tcp failed (-110) 00000100:00100000:1.0:1787699314.342108:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9d85c0fb3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3d8e19c-573a-445a-9e98-5fe67f1b3715:2359:1874531134748928:192.168.201.137@tcp:8:kworker.0 00010000:00010000:3.0:1787699314.899297:0:6059:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699315.923921:0:14500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787699316.950760:0:14499:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 74205 lines, 74205 kept, 0 dropped, 0 bad.